JupyterHub/Dask Gateway: on-hub BinderHub image builds + run-scoped clusters (PRD LCR-174; LCR-176 A+B)#162
Open
EiffL wants to merge 9 commits into
Open
JupyterHub/Dask Gateway: on-hub BinderHub image builds + run-scoped clusters (PRD LCR-174; LCR-176 A+B)#162EiffL wants to merge 9 commits into
EiffL wants to merge 9 commits into
Conversation
…art A Squashed port of the unmerged gateway-cluster-support branch (attach-only Gateway cluster branch, kubernetes container runtime, LCR-175 worker stderr forwarding, registry image resolution, design doc) plus alex/lcr-176-worker-image-deps (pinned dask stack, worker-capable Containerfile scaffold). Superseded PR #159 carried this work; this branch re-lands it as the base for the PRD run/build lifecycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
The gateway branch now mirrors the local/SLURM lifecycle instead of being attach-only: `lc run` creates a Gateway cluster with the project's worker image, scales it adaptively (1..--jobs), waits for the first worker (fail fast on unpullable images instead of hanging at zero workers), and shuts the cluster down when the run finishes — resolved decision #1 of the JupyterHub/Kubernetes PRD, and a hard requirement of the native-k8s model where picking up a freshly built image needs a fresh cluster. LIGHTCONE_GATEWAY_CLUSTER=<name> remains as the attach escape hatch for long-lived clusters: connect, never rescale, leave running on exit; the image-mismatch warning now only applies there (created clusters run exactly the requested image). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
LCR-176 Part B. New engine/binder.py drives the deployment's binderhub-service (API-only mode) from inside a user pod: - Environment ref: the last commit touching the env-defining files (Containerfile + dependency files + named COPY sources; a whole-tree COPY . is excluded — code reaches workers via the shared home, the image is the environment). Code-only commits reuse the image. - Dirty env files are committed (scoped to those paths; --no-commit to refuse instead) and pushed — build pods clone from the remote. - A root 'Dockerfile -> Containerfile' symlink is created and committed when missing (repo2docker only recognizes Dockerfile). - The build endpoint is streamed (SSE) until ready/failed; BinderHub checks its registry first, so an already-built ref is a single round-trip — cheap enough that lc run calls it every time. lc run on a kubernetes-runtime site now ensures the worker image through the same path before creating its Gateway cluster, completing the PRD user story: lc run → image up to date (rebuilt if necessary) → cluster started with it → pipeline runs → cluster culled. Off-hub kubernetes falls back to the passive registry report/verification as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
User cluster guide, lc build reference, and the api/dask_cluster page now describe the PRD lifecycle; the design doc gains §8 recording the implemented decisions (run-scoped clusters; binderhub-service resolves the §6.5 open build-path question) and marks the superseded §7.3/§7.4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
lightcone-cli | e1062e2 | Commit Preview URL Branch Preview URL |
Jul 21 2026, 06:14 PM |
✅ Eval Results
Graders✅ spec_valid (1.00) Full output |
A lightcone project wants a GitHub remote from day one — it backs the analysis up and it is what the hub's image builder clones — so lc init now actively offers to connect one, streamlined to two prompts: - Auth resolves silently from GH_TOKEN/GITHUB_TOKEN or an authenticated gh CLI; otherwise lc runs GitHub's OAuth device flow natively (stdlib urllib, no new dependencies) using the deployment-injected LIGHTCONE_GITHUB_CLIENT_ID, and hands the token to gh (gh auth login --with-token + gh auth setup-git) so one authorization powers git push and every gh workflow an agent may need. Credentials land in the user's home — the NFS volume on a hub, so they survive server restarts. - One free-form repository prompt (name | owner/name | URL): existing repos are connected, missing ones created — private supported, with the visibility default flipped to public on a hub site (the binder build pods clone anonymously) and a caveat printed when private is chosen there. - The scaffold is committed (repo-local git identity derived from the GitHub login when none is configured — fresh hub pods have none) and pushed with origin set upstream. Never fatal: failures and Ctrl-C degrade to a printed 'connect later' hint and init completes locally. Non-interactive runs use --github/--private/--public/--no-github; with no flags they get a one-line hint and never block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
From the first agent-driven project retro on the hub, the single
largest time sink was PATH shadowing: the project venv shipped its own
(older) lightcone-cli + snakemake, silently shadowing the deployment's
install and producing errors three layers from the cause. Two
structural fixes:
- lc init's venv now installs the *project's* dependencies
(requirements.txt — same list the Containerfile uses), and no longer
installs lightcone-cli at all: lc/snakemake simply never exist inside
the venv, so activation cannot shadow them. Bonus: the science code
now actually runs locally out of the box (the venv previously had
lightcone-cli but not the project's own numpy/pandas).
- lc run invokes snakemake as sys.executable -m snakemake instead of a
PATH lookup, so the driver snakemake is structurally the one matching
the running lc.
Also documents the two recipe-contract papercuts the retro hit:
{output} is a directory, and input source: paths are passed verbatim
(stage remote data locally, never a URL).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
Cluster deployments run project images as uid 1000; python:slim has no passwd entry for that uid and snakemake's getpass.getuser() call crashes at startup on every rule (found live: 'getpwuid(): uid not found: 1000' inside Gateway workers — invisible from the driver on pre-release worker CLIs). The hub also now injects USER/LOGNAME as a belt (lightcone-hub), but images should stand on their own. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
Two deletions that shrink both the CLI and the deployment contract: - The attach escape hatch (LIGHTCONE_GATEWAY_CLUSTER as a user knob) is gone: nobody iterates against sidebar-created clusters, and removing it deletes the attached-cluster path, the worker-image mismatch warning machinery, and the stale-env-var failure modes wholesale. The env var remains as the internal parent→child rendezvous only; ambient values are ignored. The gateway branch triggers on DASK_GATEWAY__ADDRESS alone. - The hub site's scratch root now defaults to $HOME (the NFS home honors flock and is mounted into worker pods at the same path), so a deployment no longer needs a shared /shared volume or the LIGHTCONE_SCRATCH env — both existed only to serve scratch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
Expert review flagged binderhub-service as heavier than the job needs on GCP (privileged dockerd DaemonSet next to user pods, dev-versioned chart, registry key in secrets, and a structural public-git-ref requirement). New lightcone.engine.cloudbuild: - Source is a tarball of the *staged build context* — the exact file set compute_image_tag hashes — uploaded to a GCS bucket. Completely git-free: no remote, no commit, no push; private projects build like public ones. The Dockerfile symlink and env-commit machinery don't apply on this path. - Image identity is the content-addressed tag (lc-<project>:<hash>) pushed to $LIGHTCONE_REGISTRY — the same identity local builds use; freshness is one registry HEAD. - Auth is the pod's Workload Identity (metadata token): zero stored credentials. Optional dedicated build SA via LIGHTCONE_BUILD_SERVICE_ACCOUNT for least privilege. - Builds run off-cluster in Cloud Build; failures fetch the log tail from the logs bucket. Backend selection (_hub_build_backend): cloudbuild when the deployment injects LIGHTCONE_BUILD_BUCKET, else the (portable) binder backend, else the passive registry report. Docs + design doc §8.5 updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the full user story of the Kubernetes/JupyterHub PRD (LCR-174): on a lightcone-hub deployment,
lc runmakes sure the worker image is up to date (rebuilding it through the hub's BinderHub service when necessary), starts a run-scoped Dask Gateway cluster with that image, runs the pipeline, and culls the cluster when it finishes — the exact same zero-configuration UX as on a laptop or a SLURM allocation. Supersedes #159 with a clean, complete branch (per review request); includes LCR-176 Parts A and B.What's in here (by commit)
Port of the Gateway groundwork (squashed from
gateway-cluster-support+alex/lcr-176-worker-image-deps, both unmerged): thekubernetescontainer runtime ("the worker pod is the container"), Gateway rendezvous-by-name through the executor plugin (a bareClientcannot dialgateway://), LCR-175 worker-stderr forwarding, hub site detection by env markers, registry image resolution, pinned dask stack + worker-capablelc initscaffold, design doc.Run-scoped cluster lifecycle (PRD decision Add Dagster execution layer for reproducible analysis pipelines #1, replaces the attach-only model): the gateway branch of
cluster_for_runnow creates a cluster with the project's image as theimagecluster option, scales it adaptively (1..--jobs), waits for the first worker (bounded byLIGHTCONE_GATEWAY_WORKER_TIMEOUT, default 600 s — an unpullable image fails loudly instead of hanging at zero workers), and shuts it down on exit. This is also what the native-k8s model requires: a Gateway cluster's image is fixed at creation, so picking up a fresh build needs a fresh cluster. There is deliberately no attach mode:LIGHTCONE_GATEWAY_CLUSTERis only the internal parent→child rendezvous (the executor rejoins the run's cluster by name —gateway://addresses can't be dialled by a bareClient); ambient values are ignored.On-hub image builds, two backends (LCR-176 Part B, resolves design-doc §6.5 — see §8.2/§8.5):
GCP Cloud Build (
engine/cloudbuild.py, preferred where the deployment injectsLIGHTCONE_BUILD_BUCKET): builds the staged build context (the exact file set the content-addressed tag hashes) off-cluster — completely git-free (no remote/commit/push; private projects build like public ones), image identity =lc-<project>:<content-hash>in$LIGHTCONE_REGISTRY, auth = Workload Identity (zero stored credentials), failures surface the build-log tail.BinderHub service (
engine/binder.py, the portable non-GCP path):env_context_paths= Containerfile + dependency files + named COPY sources, excluding whole-treeCOPY .). Code-only commits reuse the image; env edits are auto-committed (scoped to exactly those paths;lc build --no-commitrefuses instead) and pushed — build pods clone from the git remote.Dockerfile → Containerfilesymlink, maintained automatically.GET /services/binder/build/<provider>/<spec>/<sha>?build_only=true, streamed as SSE untilready/failed. BinderHub checks its registry first, so an already-built ref is one fast round-trip — cheap enough thatlc runruns the ensure step every time.JUPYTERHUB_API_TOKEN; service URL defaults tohttp://proxy-public/services/binder(LIGHTCONE_BINDER_URLoverrides). Off-hub kubernetes falls back to the previous passive registry report.Docs: user cluster guide,
lc buildreference,api/dask_cluster, and design-doc §8 recording the implemented decisions.lc initGitHub connect step (leaning into the GitHub requirement): init now actively offers to create/connect a repository — auth resolves silently fromGH_TOKEN/an authenticatedgh, otherwise a native OAuth device flow (stdlib, no new deps) using the deployment-injectedLIGHTCONE_GITHUB_CLIENT_ID; the token is handed togh(gh auth login --with-token+gh auth setup-git) so one authorization powersgit pushand all agentghworkflows, persisted in the NFS home. One free-form repo prompt (name|owner/name| URL) connects existing repos or creates new ones — private supported, visibility default flipped to public on the hub (the builder clones anonymously) with a caveat when private is chosen there. Never fatal; scripted runs use--github/--private/--public/--no-github.Deployment counterpart (lightcone-hub repo)
The deployed hub already provides everything this needs (basehub + dask-gateway + binderhub-service in API-only mode + Artifact Registry + homes mounted in worker pods). Testing surfaced one IAM gap, fixed and applied: the binder-pusher SA needed
artifactregistry.readeron thelightconerepo so build pods can resolveFROM lightcone-worker-default(commit3214c6din lightcone-hub).The notebook image now ships
ghand injects the hub OAuth app's public client id (LIGHTCONE_GITHUB_CLIENT_ID) for the device flow — verified live: a device-flow request from a user pod returns a valid one-time code with zero configuration.Known constraint: hub image builds need the repo public until
GitHubRepoProvider.access_tokenis configured on the deployment (the init step defaults visibility accordingly and warns).Testing
just checkclean: 447 tests (binder git-flow against real throwaway repos with a local bare remote, SSE parsing, provider-spec mapping, create/cull lifecycle incl. failure-path culling, stale-env-var immunity, GitHub device flow/repo bootstrap, CLI build paths), ruff + strict mypy.Live end-to-end on the deployed hub (project
EiffL/lc-gateway-e2e-test, scaffolded by this branch'slc init,Containerfile FROM lightcone-worker-default+emcee):lc build(off-hub, against the hub's public binder endpoint): created + committed theDockerfile → Containerfilesymlink, pushed, streamed the BinderHub build. Surfaced a real deployment IAM gap (build pods couldn't pull theFROMbase) as a clean error with the build-log tail — fixed in lightcone-hub, see above.lc build(in-pod, zero config):waiting → fetching → building → ready, image landed in…/binder/eiffl-2dlc-2dgateway-2de2e-2dtest-61a03a:<env-sha>(confirmed viagcloud artifacts docker images list).lc run(in-pod): image found cached in one round-trip → clusterhub.a0df6c98…created with that exact image (scheduler on the core node, worker pulling the project image on the spot pool) → recipe failure (a bug in the test recipe) reported and the cluster still culled on the failure path.Containerfile) →lc rundetected the new environment ref and rebuilt automatically before creating the cluster; a build failure at this stage surfaced the repo2docker log tail in the driver terminal.binder: waiting → fetching → building → ready→ clusterhub.0abf990f…created with the freshly built image → both rules executed in worker pods (results/baseline/main_result/result.jsonwritten fromdask-worker-0abf990f…-5zz8x,emcee 3.1.6proving the per-project env,LIGHTCONE_WORKER_IMAGE= the new binder image) →lc status: all outputs ok → cluster shut down, zero dask pods left.Known version-skew note (expected, handled): until this branch is released into the worker images, workers run the released CLI — the failure-tail forwarding to the driver terminal and the manifest
worker_imagefield stay dormant (_unpack_resultaccepts the legacy int result), while the worker-side pod log already carries the formatted rule blocks.Related
--latency-wait 60on the gateway path)gateway-cluster-support)🤖 Generated with Claude Code
https://claude.ai/code/session_01CBPsorm6U3Co7urs59FVtM
Post-review simplifications folded in (validated live on the deployment): the gateway branch is create/cull-only (attach machinery deleted); hub scratch defaults to
$HOME(no shared volume, noLIGHTCONE_SCRATCH);lc init's venv holds project deps only (no second lightcone-cli to shadow the ambient one) andlc runlaunchessys.executable -m snakemake; scaffold images carry a uid-1000 passwd entry; the hub side now needs just one image and twoLIGHTCONE_*env vars — seedocs/2i2c-deployment-requirements.mdin lightcone-hub.