Add WorkflowInterceptor extension point for named-workflow plugins#7076
Open
huangzhibo wants to merge 2 commits intonextflow-io:masterfrom
Open
Add WorkflowInterceptor extension point for named-workflow plugins#7076huangzhibo wants to merge 2 commits intonextflow-io:masterfrom
huangzhibo wants to merge 2 commits intonextflow-io:masterfrom
Conversation
Introduce a pf4j ExtensionPoint that allows plugins to intercept named workflow execution. This enables stage-level archiving and resume capabilities without modifying workflow scripts. - Add WorkflowInterceptor interface in nf-commons (ExtensionPoint) - Extract WorkflowDef.runDefault() from run() (pure refactoring) - Add interceptor lookup via Plugins.getExtension() in run() - Entry workflows (name is null) are never intercepted Signed-off-by: Zhibo Huang <huangzhibo@gmail.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: huangzhibo <zhibo90@126.com>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Relates to #5589 (checkpoint / skip-forward discussion).
Adds a minimal
pf4jExtensionPoint—nextflow.plugin.WorkflowInterceptor— that lets a plugin wrap the execution of named workflows. The core change is tiny; the actual stage-archiving / reuse logic lives in a separate plugin,nf-stage, which is the first consumer.What changes
WorkflowInterceptorinnf-commons(32 lines)WorkflowDef.run()looks up an interceptor viaPlugins.getExtension(WorkflowInterceptor); if one is registered, delegates to it with aproceedclosure that runs the original body (13 lines, extracted intorunDefault())name == null) are never interceptedTotal: +45 / -0 across 2 files.
API
The interceptor owns the invocation: it may mutate
args(e.g. clone channel inputs in place) before callingproceed(), and post-process the returnedChannelOut. Whether and whenproceed()is invoked, and how the result channels are driven, is the plugin's responsibility.Why an extension point
Issue #5589 surfaces two positions: users want stage-level skip-forward semantics; core maintainers want
-resumeleft alone and prefer pipeline authors prototype the pattern first. An extension point sidesteps the tension — it doesn't add any user-facing semantics to core, but unblocks plugin-side experimentation. If a pattern proves out in a plugin, promoting it to core remains an option later.Consumer:
nf-stagenf-stageuses this hook to archive each named workflow'semit:outputs on first run and, on reruns, replay them from a content-addressed archive when a SHA-256 digest over(stage name, static args, channel input contents)matches — nowork/dependency, no per-task hash. The mechanics (clone channel args intoargs[], letproceed()wire processes onto the clones, decide after digest compute whether to feed clones or STOP them) rely on the interceptor controlling args mutation and post-proceed channel flow, which is why the hook signature includesargs[]and aproceedclosure rather than simple before/after callbacks.Test plan
WorkflowDeftests pass (no behavior change without a plugin)nf-stageend-to-end: named sub-workflows intercepted, entry workflow not intercepted,proceed()round-tripsChannelOutcorrectly