Skip to content

Default stageOutMode to 'copy' for Google Batch executor #6917

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rhassaine wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Default stageOutMode to 'copy' for Google Batch executor #6917

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchScriptLauncher.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import nextflow.cloud.google.GoogleOpts
import nextflow.executor.BashWrapperBuilder
import nextflow.executor.SimpleFileCopyStrategy
import nextflow.extension.FilesEx
import nextflow.processor.TaskBean
import nextflow.processor.TaskRun
Expand Down Expand Up @@ -56,6 +57,13 @@ class GoogleBatchScriptLauncher extends BashWrapperBuilder implements GoogleBatc

GoogleBatchScriptLauncher(TaskBean bean, Path remoteBinDir) {
super(bean)
// Default to 'copy' stage-out mode on Google Batch. Task outputs are
// unstaged from local scratch to a gcsfuse-mounted work directory, which
// is always a cross-device operation. Using 'mv' for this fails when
// output declarations overlap (directory moved before its children) or
// when staged input symlinks resolve to the target path.
if( !bean.stageOutMode )
((SimpleFileCopyStrategy) copyStrategy).stageoutMode = 'copy'
// keep track the google storage work dir
this.remoteWorkDir = (CloudStoragePath) bean.workDir
this.remoteBinDir = toContainerMount(remoteBinDir)
Expand Down