option to save/restore relative file paths

[notestaff]

Currently, when a 'path' output of process A (e.g. param1/param2/data.csv) is passed as a 'path' input to a downstream process 'B', only the basename of the file (data.csv) is preserved; the path relative to task execution directory (param1/param2) is lost. This causes headaches whenever we adapt for Nextflow existing scripts that use paths to represent metadata. I often end up adding boilerplate code to manually save the full paths, pass them along as extra outputs, and restore them in subsequent steps. It would help if Nextflow had an option to save/restore the full relative paths automatically.

[bentsherman]

Could you write up a small test case to help us triage quickly?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pditommaso]

I sometimes found a similar need, essentially it's useful to preserve some path structure or to prevent files having the same name (but not the same path) to conflict each other when received as input.

Think also @Lehmann-Fabian had similar need.

[Lehmann-Fabian]

Hi, I created a PR to solve this: #2622. Would this be helpful to you as well? @pditommaso, maybe we can work on this PR to have such a feature?

[pditommaso]

I think we should have a more declarative approach, i.e. something like

process <NAME> {
  output: 
   path 'some/dir/*', keepDir: true 

   ...
}

Even though a better naming would be surely welcome

[Lehmann-Fabian]

When using this syntax, how can we define how many parent folders to keep? In many cases, only one or two parent dirs might be required.
While I see use cases for this, allowing closures for the path would allow much more flexibility. Since Nextflow allows Closures in many places, it would feel quite natural.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bentsherman]

Possibly related -- process input paths give you the relative path for x.name as documented here, also known as the "stage name". So it might be trivial to propagate this stage name to downstream processes, i.e. as the default stage name if you don't provide one:

process A {
  output:
  path('param1/param2/data.csv')

  script:
  // ...
}

process B {
  input:
  path data // stageAs: data.name by default

  script:
  // ...
}

workflow {
  data_ch = A()
  B(data_ch)
}

At the same time, encoding metadata in file paths in an anti-pattern, so I would hesitate to design around it.