JupyterHub on GCP: transparent Cloud Build images + run-scoped Dask Gateway clusters (LCR-174)

[EiffL]

Summary

Implements the Kubernetes/JupyterHub-on-GCP PRD (LCR-174) against the hub-deploy dask-cloudbuild deployment: on a lightcone hub, lc run makes sure the project image is up to date (building through GCP Cloud Build when needed), creates a run-scoped Dask Gateway cluster with that image, runs the pipeline in worker pods, and culls the cluster when the run finishes — the exact same zero-configuration UX as on a laptop or a SLURM allocation. Everything keys off the env contract the deployment injects into user pods (DASK_GATEWAY__*, LIGHTCONE_REGISTRY, LIGHTCONE_BUILD_BUCKET, LIGHTCONE_BUILD_SERVICE_ACCOUNT); nothing is configured by the user.

Design

One new runtime, not a new subsystem. The spec already routes everything through two seams — the container runtime (wrap_recipe, image resolution) and the cluster context (cluster_for_run) — so the k8s integration is a value flowing through existing seams:

• kubernetes runtime (engine/container.py): the worker pod is the container. wrap_recipe is a passthrough (no nested containers), and Containerfile specs resolve to registry refs. One content-addressed identity everywhere: image_identity() yields the digest that spells lc-<project>-<hash> in a local store and $LIGHTCONE_REGISTRY/lc-<project>:<hash> in the registry — so code_version, staleness, and reruns work identically on every backend.
• Build backend (engine/cloudbuild.py, ~400 lines): selected when the deployment contract is present. Freshness is a single registry HEAD on the content-addressed ref — unchanged files never rebuild, never even upload. A build tars the staged build context (the exact file set the tag hashes), uploads it to the deployment's GCS bucket, and submits a Cloud Build job that runs as the deployment's least-privilege build SA. Auth is the pod's Workload Identity via the metadata server — pure urllib, no SDK dependency, no git remote, no stored credentials. Failures surface the build-log tail.
• Gateway branch of cluster_for_run (engine/dask_cluster.py): create a cluster with the project image as the image cluster option, scale adaptively 1..--jobs, wait (bounded, default 600 s) for the first worker so an unpullable image fails loudly instead of hanging at zero workers, and shut the cluster down on exit — success or failure. Create/cull per run is PRD decision Add Dagster execution layer for reproducible analysis pipelines #1 and is also what makes image updates seamless: a Gateway cluster's image is fixed at creation. A startup assertion refuses clusters whose workers don't advertise the cpus/memory resource contract (the other silent-hang mode).
• Self-provisioned worker environment — the deployment stays near-stock: lc reads the gateway's declared cluster options and passes everything workers need through the standard environment option — the DASK_DISTRIBUTED__WORKER__RESOURCES__* scheduling contract mirrored from the declared worker_cores/worker_memory, the driver's HOME/USER/LOGNAME (passwd-less uid-1000 images crash getpass.getuser() without them), and LIGHTCONE_WORKER_IMAGE (covering the deployment-default-image case too), preserving any deployment-declared environment defaults underneath. The hub's options handler reduces to the stock 2i2c daskhub shape (cores/memory/image/environment) plus the per-user home mount — zero lightcone-specific injection server-side (hub-deploy e10088c).
• Executor rendezvous by name: gateway:// schedulers can't be dialled by a bare Client, so lc run hands the executor the cluster name (LIGHTCONE_GATEWAY_CLUSTER) and it rejoins through the authenticated Gateway API as a guest (shutdown_on_close=False).
• Output through the future result: worker pods' stdout goes to pod logs, so _run_shell now returns (exit_code, sentinel_block) and the driver prints each finished rule's block — the one channel that works uniformly across LocalCluster threads, srun workers, and Gateway pods. This retires the cross-node stdout flock entirely (one less NFS/DVS concern), and a child snakemake that dies before the rule body (missing package in the worker image) forwards a bounded raw tail instead of vanishing (PRD architecture bullet / LCR-175 semantics). Legacy bare-int results from older workers are still accepted (version skew on image-based deployments). The executor also prefixes every spawned job with cd <workdir> — spawned snakemake commands carry no --directory, and Gateway worker pods start in the image's WORKDIR, not the project (local/SLURM workers only worked by inheriting the driver's cwd).
• Site detection by env markers (engine/site_registry.py): a pod's hostname is noise; DASK_GATEWAY__ADDRESS is the signal. The jupyterhub site declares container_runtime: kubernetes and scratch_root: $HOME — no separate scratch space on the hub; home is the NFS volume shared with every worker pod, so driver and workers see one consistent snakemake state.
• Snakemake invocation shaping: on the gateway branch, --latency-wait 120 (NFS sync measured up to ~57 s on the dev hub — LCR-177) and --shared-fs-usage minus software-deployment, so the child snakemake on workers spawns plain python from the image instead of the driver's sys.executable (a path that doesn't exist in the worker container).
• lc init on the hub scaffolds a worker-capable Containerfile — the project image doubles as the worker pod image, so it carries dask/distributed/dask-gateway/snakemake/lightcone-cli pinned to the ambient hub versions (keeping the dask-gateway client in lockstep with the chart) plus a uid-1000 passwd entry so outputs stay user-owned and getpass.getuser() works (LCR-176 Part A).
• Provenance: the manifest keeps recording the declared spec, code_version hashes the registry ref (Containerfile edit → new ref → rerun), and a new additive worker_image field records LIGHTCONE_WORKER_IMAGE — provisioned into every pod by lc itself — as ground truth of what actually executed.

A spec declaring several distinct containers is rejected on this backend with guidance (the worker pod image is fixed per run — LCR-176 note); a project with no containers runs on the deployment's default image.

Out of scope (per PRD)

• Cluster lifecycle beyond one run (clusters never survive between lc run calls).
• Non-GCP build backends; general k8s support beyond the lightcone hub contract.

Relation to #162

Same PRD, different mechanism: #162 built through the BinderHub service (git-ref-based, requires committed/pushed state and a GitHub connect flow). This PR targets the newer Cloud-Build-based deployment (hub-deploy dask-cloudbuild), where builds are git-free tarball submissions — which removes the GitHub coupling and most of the surface area (~1.1k src lines vs ~4k).

Testing

• just lint clean (ruff + strict mypy), 390 tests pass (+57 new: Cloud Build control flow against mocked HTTP seams incl. failure-tail and force-rebuild, gateway create/cull lifecycle incl. failure-path culling and zero-worker timeout, self-provisioned worker environment, an image-override regression test against the real dask-gateway client (lc's explicit image must beat the ambient DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE default), executor rendezvous-by-name / workdir-prefix / legacy-result unpacking, sentinel-block forwarding, env-marker site detection, hub scratch resolution, runtime-aware status resolution, kubernetes-runtime Snakefile generation, hub-aware lc init scaffold).
• Live end-to-end validated on the deployed lightcone hub (project union2.1_lcdm_fit, user pod → Cloud Build cached-image check → run-scoped Gateway cluster with the project image → all four rules executed in worker pods with per-rule output on the driver terminal → lc verify green, manifests recording worker_image ground truth → cluster culled on exit). The live test surfaced and fixed three integration bugs: gateway pods must use the modern dask scheduler/dask worker subcommands (deployment-side, hub-deploy bfec4ab), the executor must cd into the workdir for spawned jobs (c7b18e3), and the status walker must resolve image identities runtime-aware (50508b9). A second live validation (forced full re-run) was done after the near-stock streamlining (3533b25 + hub-deploy e10088c): all rules on workers, lc status/lc verify green, manifests carrying the lc-provisioned worker_image, cluster culled — and the run automatically picked up a changed build context (new content hash → Cloud Build → fresh cluster on the new image).

Refs: LCR-174 (PRD), LCR-176 (Part A + build path), LCR-177 (latency mitigation).

🤖 Generated with Claude Code

https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
❌ Deployment failed View logs	lightcone-cli	3533b25	Jul 22 2026, 11:39 AM

[github-actions]

✅ Eval Results

Metric	Value
Score	1.00
Build complete	✅
Cost	$1.70
Turns	52
Duration	342s
lightcone-cli	0.3.8.dev7+gdc9abfa72 (dc9abfa7)
Results	Download

Graders

✅ spec_valid (1.00)
✅ all_materialized (1.00)

Full output

10:23:59 lightcone.eval.build Building lightcone-cli wheel from /home/runner/work/lightcone-cli/lightcone-cli ...
10:24:03 lightcone.eval.build Built lightcone_cli-0.3.8.dev7+gdc9abfa72-py3-none-any.whl (commit dc9abfa7)
10:24:07 lightcone.eval.sandbox Created sandbox c993f6a5-507e-4c66-af65-3db588b1ccd4 for trial build-snae-0
10:24:08 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/c993f6a5-507e-4c66-af65-3db588b1ccd4/files/bulk-upload "HTTP/1.1 200 OK"
10:24:08 lightcone.eval.sandbox Installed wheels: ['lightcone_cli-0.3.8.dev7+gdc9abfa72-py3-none-any.whl']
10:24:09 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/c993f6a5-507e-4c66-af65-3db588b1ccd4/files/bulk-upload "HTTP/1.1 200 OK"
10:24:09 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/c993f6a5-507e-4c66-af65-3db588b1ccd4/files/bulk-upload "HTTP/1.1 200 OK"
10:24:09 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/c993f6a5-507e-4c66-af65-3db588b1ccd4/files/bulk-upload "HTTP/1.1 200 OK"
10:24:10 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/c993f6a5-507e-4c66-af65-3db588b1ccd4/files/bulk-upload "HTTP/1.1 200 OK"
10:29:52 lightcone.eval.sandbox Deleted sandbox for trial build-snae-0
  snae trial 0: score=1.00 complete

lightcone-cli: 0.3.8.dev7+gdc9abfa72 (HEAD dc9abfa7)
ASTRA: 0.2.10

  Eval Results: Scores  
┏━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Task ┃     Score     ┃
┡━━━━━━╇━━━━━━━━━━━━━━━┩
│ snae │ 1.00 +/- 0.00 │
│      │ pass@k: 100%  │
└──────┴───────────────┘

   Eval Results: Cost &   
         Duration         
┏━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Task ┃ Cost / Duration ┃
┡━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ snae │      $1.70      │
│      │      342s       │
└──────┴─────────────────┘

Total: 1 trials, $1.70, 342s

Results saved to: eval-results/build-dc9abfa7/results.json

[EiffL]

Streamlining update (3533b25), live-validated: the gateway branch now self-provisions the entire worker environment through the deployment's standard environment cluster option — the DASK_DISTRIBUTED__WORKER__RESOURCES__* scheduling contract (mirrored from the gateway's declared worker_cores/worker_memory), the driver's HOME/USER/LOGNAME, and LIGHTCONE_WORKER_IMAGE (now covering the deployment-default-image case too). The matching hub-deploy change (e10088c) reduces the options handler to the stock 2i2c daskhub shape (cores/memory/image/environment) plus the per-user home mount — zero lightcone-specific injection left server-side.

Validated on the deployed hub after rolling the near-stock config: forced full re-run of union2.1_lcdm_fit — all rules executed in worker pods, lc status/lc verify green, manifests carry the lc-provisioned worker_image, cluster culled on exit. As a bonus the run picked up a changed build context automatically (new content hash → Cloud Build → fresh cluster on the new image).

🤖 Generated with Claude Code

https://claude.ai/code/session_015wg3CVQ5oYWJBCfq58Z8d2