LightconeResearch · EiffL · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026
diff --git a/docs/api/dask_cluster.md b/docs/api/dask_cluster.md
@@ -1,24 +1,48 @@
 # lightcone.engine.dask_cluster
 
 Cluster lifecycle for `lc run`. One context manager (`cluster_for_run`),
-three branches, no service to manage.
+four branches, no service to manage.
 
 Source: `src/lightcone/engine/dask_cluster.py`.
 
-## `cluster_for_run(*, verbose=False) → Iterator[str]`
-
-Yields a Dask scheduler address valid for the duration of `lc run`.
-Three branches in priority order:
-
-1. **`DASK_SCHEDULER_ADDRESS` already set** → yield as-is. We don't
-   own the cluster, so we don't tear it down.
-2. **`SLURM_JOB_ID` set** → start an in-process scheduler bound to the
+## `cluster_for_run(*, verbose=False, local_directory=None, expected_worker_image=None, max_workers=None) → Iterator[dict[str, str]]`
+
+Yields the **env overlay** the child snakemake process needs to reach
+the cluster — the parent and the executor plugin are separate
+processes, so connection info travels via environment variables.
+Four branches in priority order:
+
+1. **`DASK_SCHEDULER_ADDRESS` already set** → yield
+   `{"DASK_SCHEDULER_ADDRESS": addr}` as-is. We don't own the cluster,
+   so we don't tear it down.
+2. **Dask Gateway detected** (`DASK_GATEWAY__ADDRESS` set, e.g. a
+   JupyterHub pod) → **create a run-scoped cluster** with
+   *expected_worker_image* as its `image` cluster option, scaled
+   adaptively `1..max_workers`, and shut down on exit — the same
+   lifetime contract as the local/SLURM branches, and the mechanism
+   that makes a freshly built project image take effect (a Gateway
+   cluster's image is fixed at creation). Startup blocks until the
+   first worker is live (bounded by `LIGHTCONE_GATEWAY_WORKER_TIMEOUT`,
+   default 600 s) so an unpullable image fails loudly instead of
+   hanging at zero workers, then fails fast if workers don't advertise
+   the resource contract below. Yields
+   `{"LIGHTCONE_GATEWAY_CLUSTER": name}` — that env var is the
+   parent→child rendezvous only (Gateway scheduler addresses use a
+   `gateway://` comm scheme a bare `Client` cannot dial, so the child
+   rejoins **by name** through the authenticated Gateway API); an
+   ambient value is ignored. Requires the optional dependency:
+   `pip install lightcone-cli[gateway]`.
+3. **`SLURM_JOB_ID` set** → start an in-process scheduler bound to the
    driver's SLURM hostname (`SLURMD_NODENAME` or `gethostname()`),
    then `srun` one `dask worker` per node across the allocation.
-3. **Neither** → `LocalCluster()` sized to the local machine.
+4. **None of the above** → `LocalCluster()` sized to the local machine.
 
-The scheduler is always in-process, so its lifetime equals the run's
-lifetime: no orphaned schedulers if the driver crashes.
+Outside the Gateway branch the scheduler is always in-process, so its
+lifetime equals the run's lifetime: no orphaned schedulers if the
+driver crashes. On the Gateway branch the same contract is enforced
+server-side: the created cluster is shut down on exit
+(`shutdown_on_close=True` and the deployment's `idle_timeout` backstop
+a crashed driver).
 
 ## Resource keys
 
@@ -73,6 +97,9 @@ and keeps everything in one process tree.
 
 ## Tests
 
-`tests/test_dask_cluster.py` covers the three branches and the
+`tests/test_dask_cluster.py` covers all four branches and the
 resource-advertising contract. The SLURM branch is tested with mocked
-`subprocess.Popen` plus a stubbed `Client.wait_for_workers`.
+`subprocess.Popen` plus a stubbed `Client.wait_for_workers`; the
+Gateway branch (create/cull lifecycle, image option, worker-wait
+failure, stale-env-var immunity, resource-contract verification)
+against a fake `dask_gateway` module.
diff --git a/docs/cli/build.md b/docs/cli/build.md
@@ -14,7 +14,8 @@ lc build [OPTIONS]
 | Option | Default | Effect |
 |--------|---------|--------|
 | `--force` | off | Rebuild / re-pull even if the tag already exists locally. |
-| `--runtime {docker,podman,podman-hpc}` | resolved from `~/.lightcone/config.yaml` | Override the runtime for this build. |
+| `--runtime {docker,podman,podman-hpc,kubernetes}` | resolved from `~/.lightcone/config.yaml` (or the detected site) | Override the runtime for this build. |
+| `--no-commit` | off | On a hub: fail instead of auto-committing environment-file changes before the image build. |
 
 ## What it does
 
@@ -33,6 +34,36 @@ If the runtime is `none` (either by config or because `auto` couldn't
 find one), `lc build` prints a friendly note and exits 0. There is
 nothing to build.
 
+## On a JupyterHub deployment (`kubernetes` runtime)
+
+There is no docker in a hub pod, so `lc build` drives the deployment's
+build backend instead. Two backends exist; the deployment picks one by
+what it injects into user pods:
+
+**Cloud Build** (GCP deployments; selected by `LIGHTCONE_BUILD_BUCKET`
+being set, preferred where available). Fully git-free: the *staged
+build context* — the same file set the content-addressed tag hashes —
+is tarred and uploaded to the build bucket, built off-cluster by GCP
+Cloud Build, and pushed to `$LIGHTCONE_REGISTRY/lc-<project>:<hash>`.
+No GitHub remote, commit, or push is involved; private projects build
+exactly like public ones, and an unchanged environment is a single
+registry HEAD (no build). Auth is the pod's Workload Identity — no
+credentials anywhere. Failures surface the build-log tail.
+
+**BinderHub service** (selected by an ambient `JUPYTERHUB_API_TOKEN` /
+`LIGHTCONE_BINDER_URL`; the portable, non-GCP path). Builds from a git
+ref: `lc build` commits environment-file changes (`--no-commit`
+refuses instead), pushes (the project needs a clonable GitHub remote),
+maintains a `Dockerfile → Containerfile` symlink for repo2docker, and
+streams the build via SSE. Images are tagged by the environment
+commit.
+
+`lc run` performs the same ensure step automatically on every run, so
+running `lc build` explicitly is optional on the hub — useful to
+pre-build after a dependency change or to see the build log. With no
+backend configured, `lc build` falls back to probing the registry and
+printing off-hub publish instructions.
+
 ## Tag computation
 
 ```text

diff --git a/docs/cli/init.md b/docs/cli/init.md
@@ -36,12 +36,46 @@ universes/                    # placeholder; populate via `astra universe genera
 |--------|---------|--------|
 | `--no-git` | off | Skip `git init`. |
 | `--no-venv` | off | Skip `python -m venv .venv`. |
+| `--github [NAME\|OWNER/NAME\|URL]` | prompt (TTY only) | Connect the project to this GitHub repository without prompting; created if it doesn't exist. |
+| `--private` / `--public` | prompt (public on a hub, private elsewhere) | Visibility when `--github` creates a new repository. |
+| `--no-github` | off | Skip the GitHub connection step. |
 | `--permissions {yolo,recommended,minimal}` | `recommended` | Which `.claude/settings.json` permission tier to install. |
 
 > The historical `--target`, `--existing-project`, and `--sub-analysis`
-> flags have been removed; today's `lc init` only knows the three flags
-> above. For migrating an existing project, run `lc init` in a fresh
-> directory and use the `/lc-from-code` skill from inside Claude Code.
+> flags have been removed. For migrating an existing project, run
+> `lc init` in a fresh directory and use the `/lc-from-code` skill from
+> inside Claude Code.
+
+## The GitHub step
+
+A repository backs the analysis up, makes it shareable, and on a
+lightcone JupyterHub deployment it is what the image builder clones for
+cloud runs — so `lc init` actively offers to connect one. On a TTY it
+asks; scripted runs use the flags (no flags → a one-line hint, nothing
+blocks). The flow:
+
+1. **Auth**: an ambient `GH_TOKEN`/`GITHUB_TOKEN` or an authenticated
+   `gh` CLI is used silently. Otherwise, where a device-flow client id
+   is configured (`LIGHTCONE_GITHUB_CLIENT_ID` — injected on the hub),
+   `lc` runs GitHub's device authorization natively: enter a one-time
+   code at github.com/login/device, done. The token is handed to `gh`
+   (`gh auth login --with-token` + `gh auth setup-git`) so the single
+   authorization also powers `git push` and everything an agent might
+   do with `gh`; it lives in your home directory, which on the hub is
+   the NFS volume — it survives server restarts.
+2. **Repository**: one free-form prompt accepting a bare `name` (under
+   your account), `owner/name`, or the URL of an existing repo. If it
+   exists it is connected; if not it is created — private supported
+   (note: the hub's image builder can only clone *public* repos until
+   the deployment configures a clone token, so on the hub the
+   visibility prompt defaults to public).
+3. **Push**: the scaffold is committed (a repo-local git identity is
+   derived from your GitHub login if none is configured) and pushed
+   with `origin` set upstream.
+
+The step is never fatal: any failure or Ctrl-C leaves a fully
+scaffolded local project and prints how to connect later
+(`gh repo create --source . --push`).
 
 ## Permission tiers
 

diff --git a/docs/cli/run.md b/docs/cli/run.md
@@ -43,6 +43,26 @@ everything (Snakemake's `rule all`).
    chatter so the output reads as lightcone's, not Snakemake's. Real
    error content always passes through.
 
+## Recipe contract: outputs are directories, inputs are local paths
+
+Two things every recipe author (human or agent) needs to know at the
+point of use:
+
+- **`{output}` expands to a directory, not a file.** Create it and
+  write your artifact(s) inside (e.g.
+  `os.makedirs(out, exist_ok=True)` then write `out/result.json`);
+  opening `{output}` itself fails with `IsADirectoryError`. The
+  manifest (`.lightcone-manifest.json`) is written next to your files
+  by the runner.
+- **An input's `source:` is passed to the recipe verbatim** — it is a
+  literal filesystem path, never fetched. Stage remote data as a local
+  file first (a download script can itself be an output that later
+  recipes take as input) and point `source:` at the staged path. A URL
+  in `source:` would silently arrive at your script as a nonexistent
+  "path". Project-relative paths work on every backend because the
+  project directory is shared with (or mounted into) the workers at the
+  same location.
+
 ## Output qualification
 
 When the same `output_id` appears in multiple sub-analyses, you must

diff --git a/docs/contributing/backends.md b/docs/contributing/backends.md
@@ -6,9 +6,15 @@ of these:
 
 ## Adding a container runtime
 
-The supported runtimes are `docker`, `podman`, and `podman-hpc` (plus the
-`none` no-op). They are listed in
-`src/lightcone/engine/container.py::RUNTIMES`. To add a new one:
+The supported OCI runtimes are `docker`, `podman`, and `podman-hpc`,
+listed in `src/lightcone/engine/container.py::RUNTIMES`. Two non-OCI
+values sit outside that tuple: `none` (no isolation, recipes run on the
+host) and `kubernetes` (the Dask Gateway worker pod *is* the container;
+recipes run unwrapped inside it and images resolve against the
+deployment registry — see `REGISTRY_ENV` / `resolve_worker_image`).
+`kubernetes` is never auto-detected: a site declares it
+(`site_registry.py`, `container_runtime: "kubernetes"`) or the user
+pins it in config. To add a new OCI runtime:
 
 1. Append the binary name to `RUNTIMES` (detection priority is the tuple
    order).
@@ -23,8 +29,9 @@ The supported runtimes are `docker`, `podman`, and `podman-hpc` (plus the
 
 ## Adding a Dask cluster shape
 
-Today the cluster manager has three branches: existing scheduler, SLURM
-allocation, local. To add a fourth (for example, a custom GPU farm):
+Today the cluster manager has four branches: existing scheduler, Dask
+Gateway (JupyterHub), SLURM allocation, local. To add another (for
+example, a custom GPU farm):
 
 1. Add a branch to `cluster_for_run()` in
    `src/lightcone/engine/dask_cluster.py`.