Add NXF_DEV launcher support for automated dev builds

[ewels]

Why

Testing an unreleased Nextflow change today means waiting for an edge release, or building from source and hosting the result somewhere. That makes it slow to verify a fix on master, hard for issue reporters to confirm that a fix works for them, and effectively impossible for most users to try out functionality from an open PR.

With this change, anyone can do:

NXF_DEV=master nextflow run <pipeline>     # latest master build
NXF_DEV=pr-1234 nextflow run <pipeline>    # latest build of PR #1234

No build tools, no waiting for a release. Set inline as above, nothing persists in the environment, and the next plain nextflow invocation behaves exactly as before.

How the system works

The builds come from nextflow-io/nextflow-dev-builds, which is already live:

• A scheduled workflow there builds master and recently active PRs within ~30 minutes of a push, publishing each build as a pre-release (fat jar, standalone executable, checksums). PR code is compiled in a job with a read-only token; publishing happens in a separate job that never executes source code.
• Pointer files (latest/master, latest/pr-1234) track the newest build per channel. Old builds are pruned automatically, and builds for closed PRs are deleted.
• These builds already work with any existing Nextflow installation via NXF_BASE plus NXF_VER, since the launcher's download URL layout is matched exactly.

What this PR adds

1. NXF_DEV in the launcher: resolves the channel's latest build version from the pointer file and points NXF_BASE at the dev-builds repo. Zero overhead when unset (one bash conditional). self-update refuses to run while NXF_DEV is set, so a user cannot accidentally replace their installed launcher with a dev build.
2. dev-build-comment.yml workflow: posts a comment on each PR with copy-paste commands for testing that PR's build, updated in place on each push (never repeated). It uses pull_request_target so it also works for fork PRs; this is safe because it never checks out or executes PR code, it only reads PR metadata and the VERSION file, whose content is validated against a character allowlist before use.
3. Immediate build triggers: PR pushes (via the comment workflow) and master pushes (via dev-build-dispatch.yml) dispatch a build in the dev-builds repo right away, instead of waiting for the scheduled run. This needs the DEV_BUILDS_DISPATCH_TOKEN secret (see follow-up below). Without it, or if the dispatch fails, everything falls back gracefully to the scheduled ~30 minute build, and the PR comment states which mode applied so a misconfiguration is visible without blocking anyone.
4. Docs: an NXF_DEV entry in the environment variable reference.

Follow-up after merge

To enable immediate builds (a few minutes instead of up to 30), add a repository secret. Until then the workflows fall back to the scheduled builds automatically.

Setting up DEV_BUILDS_DISPATCH_TOKEN

1. Create a fine-grained personal access token (or use a GitHub App):
   • Resource owner: nextflow-io
   • Repository access: only nextflow-io/nextflow-dev-builds
   • Permissions: Actions: Read and write (Metadata: read-only is added automatically)
   • Expiration: per org policy; renew by updating the secret
2. In this repo: Settings, then Secrets and variables, then Actions, then New repository secret:
   • Name: DEV_BUILDS_DISPATCH_TOKEN
   • Value: the token
3. Verify: push to any open PR and check that its dev build comment says "build triggered automatically", and that a Build dev releases run appears in nextflow-dev-builds actions.

Notes

• A PR that bumps a plugin version cannot be fully tested this way, since the new plugin version is not in the registry. Core changes, the vast majority, work fine.
• Until this launcher change ships in a release, the explicit form works everywhere: NXF_BASE=https://github.com/nextflow-io/nextflow-dev-builds/releases/download NXF_VER=<build version> nextflow run <pipeline>. The PR comment shows both forms.

Companion docs PR: nextflow-io/nextflow-dev-builds#1

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	28d8c93
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/6a2fe43f06218f00088f3991
😎 Deploy Preview	https://deploy-preview-7219--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[adamrtalbot]

This would be useful for testing on remote systems, e.g. checking a fix works when launching a pipeline with Seqera Platform.

[bentsherman]

How does it handle PRs that modify core plugins?

Currently, you can either make a standard build (which won't include the core plugins because they have to be published) or you can make an all-in-one build (which doesn't support third-party plugins)

The e2e CI actually creates a launch container with the nextflow build and the core plugins baked into the plugin cache, so that we can test nf-core pipelines (which use third party plugins) against a nextflow PR build

That's the only way we have been able to get the "best of both worlds" when testing the core plugins. Another way might be to make the all-in-one build support third-party plugins...

[bentsherman]

Closing due to lack of response, please re-open when PR is actionable

[ewels]

Oops, sorry - lost track of this one. You're right that modified core plugins are the main gap in the current design.

Current state: the dev builds publish the standard one pack, so core plugins are resolved at runtime from the plugin registry as usual, at the versions listed in the build's plugins-info.txt. For a PR that modifies a core plugin that means:

• If the PR bumps the plugin version, that version doesn't exist in the registry, so the run fails with a plugin download error.
• If the PR changes plugin code without bumping the version, the runtime silently downloads the released artifact with the same version number. The user runs the dev core runtime with unmodified plugin code and gets a misleading result.

There are two ways to solve it:

1. Ship the PR-built plugins alongside the build.
   • The dev-builds repo would additionally publish a tarball of the core plugins compiled from the PR branch. The NXF_DEV launcher path would download and unpack it into a per-build directory and export it as NXF_PLUGINS_DIR. Locally present plugins take precedence over registry downloads, while third-party plugins (nf-schema etc.) still resolve from the registry as normal.
   • Essentially the same "best of both worlds" trick the e2e CI does by baking the plugin cache into the launch container, just without needing a container. It keeps the change confined to the launcher and the dev-builds workflow; no runtime behavior changes.
2. Make the all-in-one build support third-party plugins.
   • The dev builds ship the dist pack with the PR's core plugins embedded, and embedded mode would be relaxed to allow registry-downloaded third-party plugins to coexist with the bundled ones.
   • This would be a nice improvement in its own right (it'd benefit all-in-one users generally, not just dev builds), but it's more invasive and carries some risks:
     • It changes PluginsFacade embedded-mode semantics that apply to every all-in-one distribution, not just dev builds. Regressions might hit production users of the dist package.
     • Mixing bundled core plugins with downloaded third-party plugins creates a resolution/precedence question (and potential classpath/dependency conflicts) that embedded mode currently sidesteps by design.
     • It's a runtime change, so it only helps once released, whereas the launcher-side option works for testing any PR immediately.

I'll have a stab at pushing option A now as safer, but I can revert and try B if you prefer.

[netlify]

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	dc957cd
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a5d541c3251d700082a0579

[ewels]

nextflow-io/nextflow-dev-builds#2 is merged upstream ready for the latest changes here. Should be good to go now if you're happy @bentsherman

[christopher-hakkaart]

Suggestion to tighten language.