Skip to content

feat(nf-google): add configurable CLI staging transports and safer default unstaging for Google Batch#7031

Open
tomiles wants to merge 6 commits intonextflow-io:masterfrom
tomiles:fix/google-batch-gloud-staging
Open

feat(nf-google): add configurable CLI staging transports and safer default unstaging for Google Batch#7031
tomiles wants to merge 6 commits intonextflow-io:masterfrom
tomiles:fix/google-batch-gloud-staging

Conversation

@tomiles
Copy link
Copy Markdown

@tomiles tomiles commented Apr 14, 2026

This PR addresses #5653 by adding optional Google Batch stage-in/stage-out copy transports (posix, gcloud, gsutil) and aligning transfer behavior with other cloud executors while preserving backward-compatible defaults.

This design allows granular optimization of staging behavior at the process level (via stageInMode / stageOutMode), rather than forcing a single workflow-wide override. In practice, only processes that use copy for stage-in or stage-out (stage-out defaults to copy in Google Batch when unset) and have the CLI tools in their runtime environment will use the new transport path, while other modes keep existing behavior.

What changed

  • Added google.batch.stageInCopyTransport and google.batch.stageOutCopyTransport to select override for transfer transport in copy mode.
  • Added Google Batch transfer options:
    • google.batch.gcloudCli
    • google.batch.gsutilCli
    • google.batch.maxParallelTransfers
    • google.batch.maxTransferAttempts
    • google.batch.delayBetweenAttempts
  • Added/updated Google Batch bash staging helpers with CLI tool first behavior and POSIX fallback.
  • Folded in the Default stageOutMode to 'copy' for Google Batch executor #6917 behavior: default Google Batch unstaging to copy when stageOutMode is unset to avoid move failures in gcsfuse-mounted work-dir workflows (e.g. overlapping outputs/symlinked paths).

Docs

  • Updated docs/google.md with the new transport options and default unstaging rationale.
  • Updated docs/reference/config.md with all new google.batch.* config entries and defaults.

Acknowledgements

Thanks to @dmvanbeek and @rhassaine for input and hackathon discussions that helped shape this approach.

@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 14, 2026

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 6f0f786
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69f26bfc1f1e470008cc544d
😎 Deploy Preview https://deploy-preview-7031--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

tomiles added 6 commits April 29, 2026 22:37
…, gcloud, gsutil)

Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
…lized CLI transport detection

Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
…out mode

Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
Signed-off-by: Tomiles <116039+tomiles@users.noreply.github.com>
@tomiles tomiles force-pushed the fix/google-batch-gloud-staging branch from b48e886 to 6f0f786 Compare April 29, 2026 20:37
@tomiles tomiles marked this pull request as ready for review April 29, 2026 20:38
@tomiles tomiles requested a review from a team as a code owner April 29, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants