Fix module binaries PATH not injected on nextflow module run (#7087)#7088
Open
pditommaso wants to merge 1 commit intomasterfrom
Open
Fix module binaries PATH not injected on nextflow module run (#7087)#7088pditommaso wants to merge 1 commit intomasterfrom
nextflow module run (#7087)#7088pditommaso wants to merge 1 commit intomasterfrom
Conversation
When a module is launched directly via `nextflow module run`, the module main.nf is loaded as the entry script, so `ScriptMeta.isModule()` is false and `TaskProcessor.getModuleBundle()` was returning null, causing the `resources/usr/bin` directory not to be added to the task PATH. Switch the gate from `isModule()` to `getScriptPath()` so the bundle is resolved whenever the owner script has a known path. This makes the behavior of `nextflow module run` consistent with `nextflow run` plus include. The feature remains opt-in via `nextflow.enable.moduleBinaries`. Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
✅ Deploy Preview for nextflow-docs-staging ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Member
Author
|
@bentsherman what do you think about this? it may be worth adding a specific flag e.g. moduleEntry ? |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #7087.
When a module is launched directly via
nextflow module run <scope>/<name>, the module'smain.nfis loaded as the entry script, not as an included module. As a resultScriptMeta.isModule()returnsfalsefor the script that owns the process, andTaskProcessor.getModuleBundle()returnednull, so theresources/usr/bin(andresources/bin,resources/usr/local/bin) directory was never injected into the taskPATH-- even withnextflow.enable.moduleBinaries = true.This was inconsistent with
nextflow run+include { … } from './modules/...', which marks the included script as a module and gets the PATH injection correctly.Change
Single-line change in
TaskProcessor.getModuleBundle()-- switch the gate fromisModule()togetScriptPath():The bundle is now resolved whenever the owner script has a known path, regardless of whether the script was loaded as the main entry or as an included module. The feature remains opt-in via
nextflow.enable.moduleBinaries, andResourcesBundle.scan()still returns an empty bundle when noresources/directory exists, so this is a no-op for scripts without one.Why this approach
I considered three options (described in the linked issue investigation):
isModule()gate -- this PR. Smallest, most direct fix.setModule(true)-- not viable: inScriptLoaderV2setModule(true)also flipsskipEntryFlow=true, which would prevent the implicit-workflow execution thatnextflow module runrelies on. Would require a new flag (e.g.isEntryModule) -- more invasive.CmdModuleRunsynthesize a wrapper script thatincludes the module -- larger refactor.Option 1 was chosen as the minimal and semantically correct fix.
Possible side-effects and implications
The change widens the set of scripts that are eligible to contribute a
resources/bundle to the task environment. Concretely:Behavior change (intended). A standalone pipeline script
pipeline.nfthat has a siblingresources/usr/bin/(orresources/bin/,resources/usr/local/bin/) directory will now also get those directories prepended to the taskPATH, only when the user explicitly opts in withnextflow.enable.moduleBinaries = true. Previously this was silently ignored unless the script was loaded viainclude. This is arguably the correct semantics of the feature flag and matches how an end user would expect it to work.No effect when the feature flag is off.
TaskProcessor.getBinDirs()still gates the call onsession.enableModuleBinaries(), so behaviour is unchanged for the default configuration.No effect when there is no
resources/directory.ResourcesBundle.scan()returns an empty bundle when the directory does not exist (bundle/ResourcesBundle.groovy:147-148), sogetBinDirs()produces an empty list andPATHis not modified.No effect on the
include-based path. Scripts loaded as included modules continue to havegetScriptPath()set, so they keep working exactly as before.Bundle staging / fingerprinting. When the bundle is non-empty,
ResourcesBundle.fingerprint()is part of cache-key inputs in some code paths (e.g. wave/container-bundle scenarios). For a previously "non-module" script that now has a siblingresources/, the fingerprint of cached tasks could change only the first time the user opts intomoduleBinaries. Since opting in is an explicit user action this is acceptable; existing module-run / include-based pipelines are unaffected.Defensive guard kept. The check now uses
getScriptPath()rather than dropping the guard entirely, becauseScriptMeta.getModuleBundle()throwsIllegalStateExceptionwhenscriptPathis null. This preserves robustness for any code path that happens to register a script without setting its path.Test plan
./gradlew :nextflow:test --tests "nextflow.processor.TaskProcessorTest"passes (includes a new test caseshould resolve module bundle when script path is set regardless of isModulethat exercises the fix)../gradlew :nextflow:test --tests "nextflow.script.ScriptMetaTest" --tests "nextflow.SessionTest" --tests "nextflow.cli.CmdRunTest"-- existing tests for related areas still pass.nextflow -c test.config module run cellgeni/failexample --greeting 'Hello world!') succeeds with this change applied -- recommended manual verification before merge.