Skip to content

Fix cache invalidation for module binaries with Wave#6914

Open
FloWuenne wants to merge 1 commit intonextflow-io:masterfrom
FloWuenne:codex/fix-module-binaries-wave-cache
Open

Fix cache invalidation for module binaries with Wave#6914
FloWuenne wants to merge 1 commit intonextflow-io:masterfrom
FloWuenne:codex/fix-module-binaries-wave-cache

Conversation

@FloWuenne
Copy link

Summary

Fix task cache invalidation when using module binaries with Wave.

Task hashing previously considered project-level bin/ scripts but not module-level binaries under resources/usr/bin, which could cause stale cached tasks to be reused after a module binary changed.

This change makes task hashing resolve referenced executables from the task bin paths, including module binary directories, so module binary updates invalidate the cache as expected.

Closes #6128

Testing

  • Added a regression test for module binary hash changes in TaskHasherTest
  • Ran:
    • ./gradlew :nextflow:test --tests 'nextflow.processor.TaskHasherTest'

Signed-off-by: FloWuenne <flowuenne@gmail.com>
@netlify
Copy link

netlify bot commented Mar 12, 2026

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 0914b92
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69b330aebed51d0008ac8264

@pditommaso
Copy link
Member

Can you provide a reproducible test case for the problem are aiming to solve.

Module resources/bins are included via the container fingerprint, see here and here and here

@FloWuenne
Copy link
Author

FloWuenne commented Mar 13, 2026

From the issue referenced #6128, someone wrote a minimal test pipeline: jashapiro/nextflow-resume-test.

We can test this fix using the following procedure:

git clone https://github.com/jashapiro/nextflow-resume-test.git
cd nextflow-resume-test

Run once:

nextflow run . -c inspect-wave.config -with-docker

where inspect-wave.config is:

wave.enabled = true
process.container = 'community.wave.seqera.io/library/samtools:1.21--0d76da7c3cf7751c'
nextflow.enable.moduleBinaries = true

Change the module binary and rerun with resume:

touch modules/test/resources/usr/bin/hello_test.sh
nextflow run . -c inspect-wave.config -with-docker -resume

Expected result with the fix:

Cached process > hello
Submitted process > test:test_hello

Before the fix, both processes could be restored from cache even after changing modules/test/resources/usr/bin/hello_test.sh.

The included groovy test in the PR should emulate the same behaviour. I have tested the fix using a local build of Nextflow with these changes and that worked.

@pditommaso Do you prefer the fix implementation via a different mechanism? If so, can you elaborate how?

@pditommaso
Copy link
Member

If am i not wrong the test does not use wave https://github.com/jashapiro/nextflow-resume-test

@pditommaso
Copy link
Member

Ok, managed to replicate:

  • execution with wave: ok ✅
  • execution without wave: module res changes are not detected ❌

@FloWuenne
Copy link
Author

@pditommaso Thanks for replicating! Anything else missing to get this merged?

@pditommaso
Copy link
Member

Yes, this PR requires merging this first #6927

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

caching with module binaries files is not identifying changes

3 participants