Skip to content

Generate metadata CSV/TSV in modules instead of workflow body#52

Merged
ochkalova merged 26 commits into
devfrom
feat/metadata_modules
Apr 29, 2026
Merged

Generate metadata CSV/TSV in modules instead of workflow body#52
ochkalova merged 26 commits into
devfrom
feat/metadata_modules

Conversation

@ochkalova
Copy link
Copy Markdown
Contributor

@ochkalova ochkalova commented Apr 24, 2026 •
edited
Loading

Resolves #41
Resolves #44

Changes made:

  • Added local modules CREATE_ASSEMBLY_METADATA_CSV and CREATE_GENOME_METADATA_TSV and nf-corefind/concatenate to operate metadata csv/tsv files in modules for proper caching
  • Now file assigned_accessions.tsv is generated with all accessions, before we publiches separate accession files with one accession in each
  • Added versions from different tools to versions_ch
  • Snapshots updated

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/seqsubmit branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@nf-core-bot
Copy link
Copy Markdown
Member

nf-core-bot commented Apr 24, 2026

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.5.1.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 24, 2026 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit aafeff6

+| ✅ 251 tests passed       |+
#| ❔   7 tests were ignored |#
#| ❔   1 tests had warnings |#
!| ❗  15 tests had warnings |!
Details

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • local_component_structure - rna_detection.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - genome_evaluation.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • nextflow_config - Config variable ignored: params.input
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/nf-core-seqsubmit_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-seqsubmit_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-seqsubmit_logo_dark.png

❔ Tests fixed:

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.1
  • Run at 2026-04-28 13:37:13

@ochkalova ochkalova changed the title Feat/metadata modules Generate metadata CSV/TSV in modules instead of workflow body Apr 24, 2026
ochkalova added 20 commits April 27, 2026 14:55
@ochkalova
refactor metadata files creating modules
6340b4b
@ochkalova
add find_concatenate module
7cb2880
@ochkalova
patch find/concatenate
c6b7fbc
@ochkalova
fix version statement for GENERATE_ASSEMBLY_MANIFEST module
8304fbf
@ochkalova
add FIND_CONCATENATE to assemblysubmit and update tests
f217ac8
@ochkalova
add FIND_CONCATENATE to genomesubmit and update tests
48abbfd
@ochkalova
add additional inputs instead of params usage subwf
badfb2b
@ochkalova
add additional inputs instead of params usage in ENA_WEBIN_CLI_WRAPPER
47b8698
@ochkalova
add additional inputs in GENOME_UPLOAD
c410d5e
@ochkalova
add additional inputs in GENERATE_ASSEMBLY_MANIFEST
708cb1e
@ochkalova
update tests
fd698fd
@ochkalova
fix checkm2_db definition
689259a
@ochkalova
remove --upload_force param because it's unnecessary for a user
74fa995
@ochkalova
do not reference params in ASSEMBLYSUBMIT and GENOMESUBMIT
1371dec
@ochkalova
add test_upload parameter to REGISTERSTUDY and GENERATE_ASSEMBLY_MANI…
93439b4 
…FEST processes
@ochkalova
rename checkm2_db_zenodo_id to checkm2_db_download_id for consistency
ff2d46d
@ochkalova
add missing REGISTERSTUDY input
7a6030c
@ochkalova
rename webincli_submit to webincli_mode
4cf9a0e
@ochkalova
add CREATE_ASSEMBLY_METADATA_CSV module, add versions tracking for al…
4530c73 
…l modules
@ochkalova
add CREATE_GENOME_METADATA_TSV module
0b2d785
@KateSakharova KateSakharova mentioned this pull request Apr 28, 2026
Merged
@ochkalova ochkalova force-pushed the feat/metadata_modules branch from 75461a7 to c7b2603 Compare April 28, 2026 13:20
@ochkalova ochkalova marked this pull request as ready for review April 28, 2026 13:28
@KateSakharova KateSakharova self-requested a review April 28, 2026 15:32
KateSakharova
KateSakharova approved these changes Apr 28, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@KateSakharova KateSakharova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can merge it. I've rebased my multiqc fixes on top of that branch already. If something pop up I will fix there :)

@ochkalova ochkalova merged commit d8beefc into dev Apr 29, 2026
21 checks passed
@KateSakharova KateSakharova deleted the feat/metadata_modules branch May 15, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@KateSakharova KateSakharova KateSakharova approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add modules for CSV/TSV metadata generation Track versions in all workflows of the pipeline

3 participants

@ochkalova @nf-core-bot @KateSakharova