Add cluster metrics viz

[dbaku42]

Description

This PR adds two new modules:

• cluster_metrics: Computes standard clustering quality metrics (Silhouette score, Calinski-Harabasz index, Davies-Bouldin index) and performs a k-sweep analysis.
• cluster_viz: Generates 2D visualizations (PCA, UMAP, t-SNE) colored by cluster label and exports the coordinates as TSV files.

Both modules are designed to work together with the existing clustering module (which performs KMeans/DBSCAN on PLINK2 PCA results).

Features

• Full support for conda via environment.yml
• Comprehensive nf-test (normal + stub tests)
• Clean output channels with proper naming
• Optional plots (PNG) for visualization

Author

• Donald Baku (@dbaku42)

Related modules

• Related to: #11337 (clustering)

---

This completes the core components of the snpclustering subworkflow.

[SPPearce]

Um, you need to rebase these changes on the current master, as there was a bulk update (and you have a massive conflict ;) ).
Ping me when you have done that.

[SPPearce]

Hmm, this is an entirely custom set of modules?
This needs discussion as to where they live.
They could potentially be in custom/

[dbaku42]

Thanks @SPPearce, that makes sense.

I’ve moved both custom Python-based modules under modules/nf-core/custom/, updated the internal paths and tests, fixed the author typo, and refreshed the nf-test snapshots.

Both module tests now pass locally for cluster_metrics and cluster_viz.

[SPPearce]

I think it'd need to be custom/clustermetrics and custom/clustervisualiation (no underscores allowed).
You'll need to use template directly, rather than referring to ${projectDir}/modules/nf-core/custom/cluster_metrics/templates/cluster_metrics.py.

[dbaku42]

Thanks @SPPearce — I renamed the custom modules to remove underscores and updated both modules to use ${moduleDir} instead of hardcoded ${projectDir} paths. I also updated the tests and snapshots accordingly.

[SPPearce]

You can't use moduleDir at all in the main.nf, it won't work on cloud systems as the python script won't be there.
You need to use a template, like these modules.

[SPPearce]

You could make the docker/singularity containers via Seqera Containers rather than needing a custom image in the nf-core quay.io.

[pinin4fjords]

Hi @dbaku42, thanks for the contribution. AI-assisted review (Claude, on behalf of @pinin4fjords). Some overlap with @SPPearce's earlier feedback. Beginner-friendly notes plus suggestion blocks you can apply directly.

The biggest fix is the templates/ mechanism. Right now your script: block runs python3 ${moduleDir}/templates/cluster_metrics.py, which is just running a normal Python script that happens to live in templates/. No templating is happening. This works on your laptop because ${moduleDir} resolves to a local path, but it breaks on cloud/HPC executors because that folder is not staged into the work directory the task runs in.

Nextflow's template directive does two things:

1. Stages the file into the task work directory automatically (so it works on any executor).
2. Runs the file through Groovy GString interpolation before staging, so ${features}, ${task.ext.prefix}, etc. get substituted into the Python source. This replaces argparse entirely.

Because the file goes through Groovy interpolation, a few Python source patterns need adjusting (\n -> \\n, r"\s+" -> r"\\s+", etc., I verified this locally). I have flagged the specific spots inline. See modules/nf-core/custom/tx2gene/templates/tx2gene.py for the canonical shape.

Two scope questions for discussion:

• The *_selected.json "best k" output bakes a decision into the module, and the sweep refits KMeans regardless of what method produced the input labels. Could the module just emit the k-sweep table and let the consuming pipeline pick?
• cluster_viz does PCA + UMAP + t-SNE + plotting + TSV export, and the PCA piece is essentially a re-plot of upstream PLINK eigenvec output. Consider dropping PCA plotting from this module and tightening to the embedding methods that genuinely belong together.

The clustervisualiation directory name looks like a typo for clustervisualization; worth fixing while you're renaming things.

[dbaku42]

Hi @pinin4fjords , I'm stuck on a snapshot mismatch issue and would appreciate some guidance. The tests for CUSTOM_CLUSTERMETRICS and CUSTOM_CLUSTERVISUALIZATION are failing because the output files (test_metrics.tsv, test_k_sweep.csv, test_selected.json) produce different MD5 hashes in CI compared to my local environment.

The root cause seems to be that my local snapshots were generated using the Wave container, while the CI runs tests using conda (as shown in the logs: Creating env using conda: .../environment.yml). The two environments resolve slightly different package versions, leading to numerically different outputs from scikit-learn/pandas.

Thanks in advance! 🙏

[pinin4fjords]

Hi @dbaku42, sorry for the volume of direct commits on your branch (AI-assisted by Claude on my behalf) - given the back-and-forth pattern of the last few rounds it felt like the fastest way to land the modules in a clean, spec-aligned state. Please review and push back on anything that doesn't sit right; happy to revert specific bits if you disagree with a choice. Highlights of what changed and why:

• Containers rebuilt on Seqera Containers via Wave to match the actual environment.yml for each module. nf-core uses Biocontainers or Seqera Containers; for multi-package Python envs like these, Seqera Containers via the wave CLI is the right call. Singularity branch is the https://community-cr-prod.seqera.io/.../data blob URL (not docker://).
• ${...} substitutions in clustering.py: ruff lints the source as Python, so KMeans(n_clusters=${n_clusters}, ...) fails. Moved substitutions to locals at the top of main() and use the locals from then on.
• versions.yml writes via yaml.dump (with pyyaml added to the envs), since adding deps is cheap now that we're on Wave. Stub heredocs use importlib.metadata.version(...) to avoid import side effects (import umap triggers numba JIT cache writes that fail under read-only singularity).
• Test portability: KMeans inertia wobbles a few ULPs across CPUs (BLAS reduction order), so the clustering test rounds it in the assertion only. Added a DBSCAN test case so both algorithm branches are exercised.
• ext.args plumbing in cluster_metrics.py (--k-min, --k-max) and cluster_viz.py (--umap-neighbors, --tsne-perplexity), parsed via argparse + shlex.split("$task.ext.args") inside the template - matches the spec that optional non-file args go via ext.args. Same approach as custom/matrixfilter uses for R.
• Output filenames unified to ${prefix}.<descriptor>.<ext> across all three modules (was mixed dot/underscore), matching the spec example and the existing custom/tx2gene / custom/matrixfilter pattern.
• Simplifications: removed the PLINK-aware _normalise_id_column / multi-mode load_clusters from cluster_metrics.py and cluster_viz.py (both already require sample_id + cluster and the clustering module emits exactly that). Replaced the positional-index alignment with load_features().join(load_clusters(), how="inner"). Replaced the 45-line manual eigenvec parser with pd.read_csv. Reverted a stray .gitignore change unrelated to this PR.

One blocker before this can land: tests/data/ files can't live here. nf-core stores all module test data in nf-core/test-datasets on the modules branch, referenced via params.modules_testdata_base_path. See custom/basicpy for an example. To unblock:

1. PR your five test files to the modules branch of nf-core/test-datasets under something like genomics/popgen/clustering/.
2. Once it merges, update each tests/main.nf.test to use params.modules_testdata_base_path + ... and delete the tests/data/ directories.
3. Re-record snapshots and push.

Marking as Changes Requested for that, and resolving the previous threads since they're addressed by the recent commits. Apologies again for the heavy hand - shout if anything needs reverting.

Local workflow to catch the easy stuff before pushing in future:

prek run --all-files
nf-test test --profile docker --update-snapshot modules/nf-core/custom/clustering/tests/main.nf.test
nf-test test --profile docker --update-snapshot modules/nf-core/custom/clustermetrics/tests/main.nf.test
nf-test test --profile docker --update-snapshot modules/nf-core/custom/clustervisualization/tests/main.nf.test

[dbaku42]

Hi @pinin4fjords,

thank you so much for all the work you did directly on the branch (and for the super clear explanations)!
I've reviewed all the commits and they look excellent: the Wave/Seqera containers, correct use of the template directive, task.ext.args, unified file naming, versions.yml with pyyaml + importlib.metadata, the Singularity/Numba fixes, etc. I have no objections at all — on the contrary, thank you for speeding up the process so much.

Regarding the remaining blocker:

• I'm preparing right now a PR on nf-core/test-datasets (branch modules) to upload the 5 test files under genomics/popgen/clustering/ (as you suggested).
• As soon as that PR is merged, I'll update the tests/main.nf.test files for both modules, delete the tests/data/ folders, re-snapshot and push the changes here.

I'll let you know as soon as the test-datasets PR is opened (and I'll tag you).
If you'd like me to use a slightly different path or rename anything in the test files, just let me know!

Thank you again for the huge help ❤️
(By the way, the modules look really great now)

@nf-core/modules

[dbaku42]

Hi @pinin4fjords,

I've just opened the PR for the test data on nf-core/test-datasets:
nf-core/test-datasets#2051

As soon as it's merged I'll update the two main.nf.test files, remove the local tests/data/ folders, re-snapshot and push here.

Thanks again for all the help!