From 514da9bac9f3b19a1c7f80e763f9a3c4113cd9a3 Mon Sep 17 00:00:00 2001 From: Jannes Stubbemann Date: Sat, 18 Jul 2026 08:58:47 +0200 Subject: [PATCH] docs: record ADR 0009 (etcd hot path) and reconciler-sharding spec ADR 0009 settles whether Kubernetes is the wrong substrate for sandboxes against the Mitos claim path. Warm pools already remove kube-scheduler and the readiness watch already removes create latency, so the residual is the per-sandbox etcd write throughput. The decision: keep the Sandbox CR as the durable record, measure the control-plane ceiling (issue #15 item 2) before decoupling, and if it binds, shard the reconciler (Option A) before ever moving the claim into the gateway (Option B, which needs its own ADR and a named security review). Adds the ready-to-execute Option A design spec and indexes the ADR. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Jannes Stubbemann --- ...ndbox-lifecycle-state-and-etcd-hot-path.md | 193 ++++++++++++++++++ docs/adr/README.md | 1 + .../2026-07-18-reconciler-sharding-design.md | 143 +++++++++++++ 3 files changed, 337 insertions(+) create mode 100644 docs/adr/0009-sandbox-lifecycle-state-and-etcd-hot-path.md create mode 100644 docs/superpowers/specs/2026-07-18-reconciler-sharding-design.md diff --git a/docs/adr/0009-sandbox-lifecycle-state-and-etcd-hot-path.md b/docs/adr/0009-sandbox-lifecycle-state-and-etcd-hot-path.md new file mode 100644 index 00000000..23788695 --- /dev/null +++ b/docs/adr/0009-sandbox-lifecycle-state-and-etcd-hot-path.md @@ -0,0 +1,193 @@ +# ADR 0009: the Sandbox CR stays the durable record; scale the reconciler before moving the claim into the gateway + +Status: proposed (2026-07-18) +Issue: #15 (controller-path benchmarks; the sustained claims/sec and density +curve). Related: `internal/saas/controlplane` (the hosted create path, +`forward.go`, `readywatch.go`), `internal/controller/sandboxclaim_controller.go` +(`reconcileHuskClaim`), `internal/controller/huskpod.go` (`markHuskPodClaimed`, +`unmarkHuskPodClaimed`, `selectDormantHuskPod`), `docs/husk-pods.md`, +`docs/failure-gc.md` (status elision, TTL GC), `docs/threat-model.md`, +`BENCHMARKS.md` (#15 item 2, and the CI control-plane datapoint added with this +ADR). Motivating external critique: Modal, "Scaling to 1 million concurrent +sandboxes in seconds" (the argument that running one Kubernetes control-plane +transaction per sandbox does not scale). + +## Context + +The external critique is narrow and, on its own terms, correct: the Kubernetes +control plane is a poor per-sandbox transactional store at very high create +rates. The named bottlenecks are real (kube-scheduler is `O(n x p)` and +serialized by default, roughly 100 pods/sec in a live cluster; etcd write +throughput is single-node bound and not shardable within a keyspace; per-object +watch and reconcile add load per object). The question this ADR settles is how +much of that lands on Mitos, and what, if anything, to change. + +What Mitos already does that sidesteps the worst of it: + +- Sandboxes are not pods. Warm husk pods are pre-scheduled and pre-admitted, so + the claim hot path is `selectDormantHuskPod` + `markHuskPodClaimed` (an + optimistic-locked label patch) + mTLS activate, NOT a kube-scheduler decision + (`docs/husk-pods.md`, `sandboxclaim_controller.go:782` reconcileHuskClaim). The + kube-scheduler `O(n x p)` ceiling is off the claim path by construction. +- The exec / files / run_code data path never touches the API server: SDK -> + podIP:sandboxPort -> vsock -> guest agent. +- Create LATENCY is already solved. The create response is delivered by a WATCH + on the single Sandbox (`readywatch.go`), not a poll tick. The code records that + the tick quantization it replaced was the dominant cost: p50 545ms client + observed versus 6-40ms on-node activate (`forward.go` pollReady doc comment). + So the remaining question is throughput, not latency. +- Reconcile churn is already bounded. `writeClaimStatusIfChanged` deep-compares + before every status write so a re-reconciling claim does not churn etcd or + re-trigger its own watch (`docs/failure-gc.md`), and finished objects are TTL'd + out of etcd. + +What remains coupled: every create, fork, and terminate is a Sandbox CR write +plus a watch plus a reconcile that performs a small number of status writes. +Under the elision, the steady state is on the order of two etcd writes per +sandbox (the create and the terminal Ready status). That is real per-sandbox +control-plane load, and it is the thing the external critique points at. + +The honest gap: we have not measured whether that residual binds. etcd sustains +thousands of small writes per second; warm pools already removed the scheduler; +the watch already removed the latency. Whether two writes per sandbox plus a +watch is a wall at Mitos's target scale is an open, measurable question, and it +was unmeasured. Deciding to move the claim out of the control plane before +measuring would be optimizing a bottleneck we have not shown exists, while taking +on real risk (below). This ADR therefore ties the decision to a number. + +The measurement is now wired: `BENCHMARKS.md` #15 item 2 and the `kind-e2e` +control-plane throughput step produce a real achieved-claims/sec datapoint on +every CI run (mock engine, single node: the etcd/apiserver/reconciler ceiling, +not hardware density), and `bench/sustained-claims-throughput.sh` traces the full +density curve on the reference node by sweeping the arrival rate. + +## Decision drivers + +1. Do not weaken the single-writer invariant. Claim correctness rests on exactly + one writer of the `mitos.run/claim` label per pool, bounded today by leader + election (`reconcileHuskClaim` doc comment: "Leader election (one active + reconciler) is what bounds concurrent claiming"). The atomic claim commit + `markHuskPodClaimed` is optimistic-locked and race-safe, but the RELEASE path + `unmarkHuskPodClaimed` is a plain `MergeFrom` with NO optimistic lock, safe + only because "it is the claim that stamped the label releasing it" + (`huskpod.go`). A second concurrent claimer breaks that assumption. +2. Do not move the cross-tenant activation and mTLS surface into the gateway + without strong, measured reason. Activation delivers secrets and the + per-sandbox token over mTLS and is a security-sensitive path + (`CLAUDE.md` names `internal/controller` claim/activation and future + token/attenuation code for a named human reviewer). +3. Preserve the create contract: return `{id, endpoint, phase, token}` on create. +4. Prefer horizontal scaling that mirrors the external system's own answer + (a fleet of stateless schedulers) without adopting its consistency tradeoffs + where we do not have to. + +## Options considered + +### A. Keep the CR as the record; scale the reconciler horizontally (recommended lever if the ceiling binds) + +Keep the Sandbox CR as both the claim and the durable record. Remove the SERIAL +reconciler ceiling by sharding Sandbox objects across controller replicas +(consistent-hash or label-partitioned work queues, one lease per shard) so +reconcile throughput scales with replicas, the way the external system scales +with stateless scheduling servers. etcd stays the store; the single-writer +invariant is preserved because each shard is the sole writer of its own objects. + +- Pro: no security-surface change; no new trust path; the claim/activation stays + in the leader-elected (now shard-owned) controller; the fix is horizontal and + boring. +- Pro: directly attacks the measured bottleneck if it is reconcile serialization + rather than raw etcd write cost. +- Con: does not reduce the ~2 etcd writes per sandbox. If the wall is etcd write + throughput itself (not reconcile serialization), sharding the reconciler does + not move it; that case needs etcd-side work (a dedicated events/leases store, + or fewer writes per sandbox) or Option C. +- New invariant to prove: shard ownership must guarantee no two replicas own the + same Sandbox concurrently, or the single-writer claim invariant breaks. + +### B. Inline claim AND activate in the gateway; write the CR asynchronously + +The gateway does `selectDormantHuskPod` + `markHuskPodClaimed` + mTLS activate +inline and returns the endpoint, writing the Sandbox CR asynchronously as a +record. Maximum decoupling: etcd is fully off the create critical path. + +- Pro: the strongest form of "no data store on the critical path." +- Con: the gateway needs husk mTLS client credentials, a new secret distribution + and a materially larger blast radius for the cross-tenant boundary. +- Con: two writers of the claim label. The commit is safe (optimistic lock), but + `unmarkHuskPodClaimed` is not optimistic-locked, the token Secret write assumes + single ownership, and Sandbox status writes are un-serialized; all three are + documented single-writer assumptions that a concurrent claimer corrupts. +- Con: a gateway that claims a pod then dies before writing the CR leaks a + claimed pod with no owning object; needs orphan-claim GC. +- This is the highest risk and the largest security-surface move. + +### C. Reservation split: gateway claims inline, controller adopts and activates + +The gateway does ONLY the race-safe `markHuskPodClaimed` inline and writes the CR +referencing the already-claimed pod; the reconciler ADOPTS the pre-claimed pod +and performs activation, token minting, and status. Activation, mTLS, token, and +status stay single-writer in the controller. + +- Pro: removes warm-pod selection churn from the reconcile loop and lets create + commit a reservation immediately, without giving the gateway mTLS or activation. +- Con: still writes the CR (as the record); create still waits on the reconciler + to activate (via the watch), so it does not fully take etcd off the path. +- Con: still two writers of the claim label; requires an optimistic-locked + release and orphan-claim GC (a claim label whose owning CR does not exist). +- Middle risk; a fallback if A is measured insufficient and the etcd write itself + is not the wall. + +## Decision + +1. The Sandbox CR remains the durable lifecycle record. We do not adopt Option B + now. Moving the claim and activation into the gateway trades a measured, + bounded security posture for an UNMEASURED throughput gain and is not + justified until the CR-per-create path is shown to be the binding constraint. +2. Measurement gates the next step. The control-plane ceiling is now produced in + CI and sweepable on the reference node (#15 item 2). No lifecycle-decoupling + code lands until that number shows the CR-per-create path is the binding + constraint at a concrete target scale. +3. If it binds, Option A (shard the reconciler) is the first lever, because it + scales throughput horizontally without weakening the single-writer invariant + or moving the security boundary. Option C is the reserved fallback if A is + insufficient AND the wall is not raw etcd write throughput; Option B is not + pursued without a separate ADR and a named security review. +4. Any decoupling work ships behind a flag, default off, with the CR-synchronous + create path remaining the default, and lands with its threat-model delta and a + named human reviewer for the security-sensitive claim/activation path. + +## Threat-model delta (applied when code lands, not now) + +This ADR moves no code, so `docs/threat-model.md` is unchanged today. The deltas +below are recorded so the reviewer of any implementation PR knows exactly what +the chosen option moves, per the CLAUDE.md rule that the threat model updates in +the same PR as the surface change. + +- Option A: sharding must guarantee exactly-one-owner per Sandbox (a per-shard + lease). New row: a sharding bug that lets two replicas own one Sandbox breaks + the single-writer claim invariant; the mitigation is lease-fenced shard + ownership plus the existing `markHuskPodClaimed` optimistic lock as the + backstop. No new cross-tenant surface. +- Options B and C: the gateway gains the ability to stamp `mitos.run/claim` on + husk pods, so it must re-check pool ownership (`huskPodOwnedByPool`) and the + namespace boundary exactly as `selectDormantHuskPod` does today; the release + path must become optimistic-locked or claim-name-guarded; orphan-claim GC + becomes required or warm capacity leaks. Option B additionally gives the + gateway husk mTLS client credentials: a new secret to distribute and a larger + blast radius for the cross-tenant boundary, which is why B needs its own ADR. + +## Consequences + +- The near-term deliverable for "does the Kubernetes bet scale" is the NUMBER, + not a rewrite. #15 item 2 is now exercised on every CI run and sweepable on the + reference node; that is what converts the external critique from an unrebutted + blog argument into a measured curve for Mitos's actual path. +- The honest public position is unchanged and now measurable: warm pools remove + the scheduler, the watch removes create latency, elision and TTL GC bound + churn, and the residual per-sandbox etcd write is a known, bounded cost whose + scaling ceiling we now publish rather than assert. +- Where Mitos actually beats the cold-start-and-schedule model is the fork axis + (node-local live fork, CoW density), which is orthogonal to this ADR and not in + scope here; this ADR only settles the control-plane substrate question. +- If measurement shows the ceiling binds, the path is pre-decided (A, then C), + behind a flag, with a threat-model delta and a named reviewer. diff --git a/docs/adr/README.md b/docs/adr/README.md index 63b7e068..0d27a1c4 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -70,6 +70,7 @@ docs/facade-conformance.md. See ADR 0000 for the full numbering note. | [0006](0006-husk-netadmin-egress-firewall.md) | The husk-pod NET_ADMIN capability for in-pod egress firewalling | proposed | One scoped `NET_ADMIN` capability, in the pod's own netns, as the minimal control for default-deny husk egress plus a metadata block. | | [0007](0007-api-v2-three-noun-consolidation.md) | API v2 consolidates four kinds to three nouns (Pool, Sandbox, Workspace) | accepted | Folding `SandboxTemplate` into the pool and `SandboxFork`+`SandboxClaim` into `Sandbox`; the v1alpha1 to v2 conversion contract; discharging ADR 0001's deferred rename as one breaking migration. | | [0008](0008-forkd-non-privileged-jailer.md) | forkd runs non-privileged with the jailer enabled | accepted | Dropping forkd's `privileged: true` for an explicit, audited builder capability set with the per-VM jailer ENABLED and `/dev/kvm` from the device plugin (#352); the residual is uid 0 + CAP_SYS_ADMIN, not a privileged container. | +| [0009](0009-sandbox-lifecycle-state-and-etcd-hot-path.md) | The Sandbox CR stays the durable record; scale the reconciler before moving the claim into the gateway | proposed | Warm pools already remove kube-scheduler and the watch already removes create latency, so the residual per-sandbox etcd write is measured (#15 item 2 in CI) before any decoupling; if the ceiling binds, shard the reconciler (Option A) before moving the claim/activation into the gateway. | ## Residual ADRs and the compliance claim-language rule diff --git a/docs/superpowers/specs/2026-07-18-reconciler-sharding-design.md b/docs/superpowers/specs/2026-07-18-reconciler-sharding-design.md new file mode 100644 index 00000000..240cc3a5 --- /dev/null +++ b/docs/superpowers/specs/2026-07-18-reconciler-sharding-design.md @@ -0,0 +1,143 @@ +# Reconciler sharding (Option A of ADR 0009): design spec + +Status: ready to execute, gated on measurement (do not implement until the #15 +item 2 curve shows a single reconciler is the binding control-plane constraint). +Decision record: `docs/adr/0009-sandbox-lifecycle-state-and-etcd-hot-path.md`. +Issue: #15. Grounding code: `internal/controller/sandboxclaim_controller.go` +(`reconcileHuskClaim`), `internal/controller/huskpod.go` (`markHuskPodClaimed`, +`unmarkHuskPodClaimed`, `selectDormantHuskPod`), `cmd/controller/main.go` +(manager + leader election wiring), `internal/controller/node_registry.go`. + +## Problem + +The `SandboxReconciler` runs as one leader-elected reconcile loop. Within the +active leader, controller-runtime drains a work queue with +`MaxConcurrentReconciles` workers, so the control-plane claim throughput ceiling +is one leader's aggregate reconcile rate. Leader election gives us HA (one active +leader, standbys idle), not horizontal throughput: the standbys do no work. When +the #15 item 2 curve shows achieved claims/sec plateauing below the arrival rate +with the leader CPU-bound and its work queue growing, that single-leader ceiling +is the wall, and this spec is how we lift it. + +The hard constraint: the claim correctness invariant. Exactly one actor may +successfully claim a given warm husk pod. Today that is bounded by "one active +reconciler" (leader election). Any horizontal scheme must not let two reconcilers +corrupt a claim. + +## Key insight: sharding is load distribution, not correctness + +Correctness does NOT have to rest on perfect single-ownership of a Sandbox. The +pod-claim commit is already the true arbiter: + +- `markHuskPodClaimed` (`huskpod.go`) patches the `mitos.run/claim` label under + `client.MergeFromWithOptimisticLock`. Two actors that select the same dormant + pod both attempt the patch; the API server lets exactly one win and the other + gets 409, which the caller handles by pending + requeue (`HuskPodRaced`). This + holds no matter how many reconcilers race. + +So if the RELEASE path and the token/status writes are hardened against a second +writer, then sharding becomes a pure LOAD-DISTRIBUTION optimization: a brief +two-owner window (for example during a shard-count resize) is merely wasted work +(a 409 race), never corruption. This is what makes Option A cheap and safe, and +it is why we do NOT need a perfect exactly-one-owner lease as a correctness +mechanism. The single hardening prerequisite: + +- Harden `unmarkHuskPodClaimed`. Today it is a plain `client.MergeFrom` with NO + optimistic lock, justified by "it is the claim that stamped the label releasing + it." With two potential claimers that justification fails: a stale owner could + blindly clear a `mitos.run/claim` label a different claim just stamped. Fix: + make the release a claim-name-guarded, optimistic-locked patch. It releases the + label ONLY if the label still names THIS claim, else it is a no-op. This is a + small, self-contained correctness improvement worth landing first and on its own + merits, independent of sharding. + +## Design + +Three landable steps, in order. Steps 1 and 2 are safe to land before the gate +opens; step 3 (turning sharding ON) is what waits on the measurement. + +### Step 1: harden the claim release (correctness, land anytime) + +Change `unmarkHuskPodClaimed` to a claim-name-guarded optimistic-locked patch as +above. Add a unit/envtest that a release whose pod label names a DIFFERENT claim +is a no-op (does not un-claim the other claim's pod). + +### Step 2: hash-in-predicate shard filtering (default single shard = today) + +Each reconciler owns a set of hash ranges over `shard(namespace/name)`. It +reconciles object O only when `shard(O)` is in its owned set; otherwise the +predicate drops the event before it enters the work queue. No object is mutated +(no shard label, no webhook), so there is nothing to rewrite on a resize and no +extra etcd write per sandbox. + +- `shard(key) = fnv32a(namespace + "/" + name) % shardCount`. +- Membership is STATIC in v1: `--shard-count N` (default 1) and `--shard-index i` + (default 0), where `i` comes from the pod's StatefulSet ordinal via the downward + API. `shardCount = 1` reproduces today's behavior exactly (one owner of + everything), so the flag defaults to a no-op. +- Wire the filter as a `predicate.Predicate` on the `SandboxReconciler`'s + `For(&v1.Sandbox{})` watch AND as a cache `ByObject` field/label restriction if + cheap; the predicate is authoritative. Fork children and the pool/workspace + watches shard by the OWNING Sandbox's key so a Sandbox and its forks land on the + same shard (locality: a live fork is node-pinned to its source anyway). +- Leader election changes shape: with a StatefulSet of N controllers, each pod is + the sole active reconciler FOR ITS SHARD. Either run per-shard leader election + (a lease per shard index) so each shard still has HA standbys, or run the + controllers as a StatefulSet where each ordinal is its own singleton. v1 uses a + per-shard lease keyed by shard index, so HA is preserved within a shard. + +### Step 3: turn it on (gated) + +Flip `shard-count > 1` and scale the controller StatefulSet only when the +measurement (below) says the single-leader reconcile rate is the binding +constraint. Because sharding is load distribution and the lock is the arbiter, a +resize from N to N+1 (which remaps object->shard) needs no stop-the-world: during +the brief window where old and new owners disagree on a few objects, the +optimistic-locked claim + guarded release make the overlap harmless (wasted 409 +races, no corruption). A rolling resize with the standard lease TTLs is +sufficient. + +## Test plan (envtest, all offline) + +- Release guard (step 1): a pod whose `mitos.run/claim` names claim B is not + released by claim A's `unmarkHuskPodClaimed`; claim A's own release is a no-op + once the label no longer names A. +- Shard partition (step 2): two `SandboxReconciler`s over one envtest apiserver, + `shard-count=2`, indices 0 and 1. Assert each reconciles only keys whose hash + matches its index (observe reconcile invocations), every Sandbox still reaches + Ready, and no warm pod is double-claimed. +- Pathological two-owner (correctness backstop): two reconcilers BOTH index 0 + (simulating a resize overlap) claiming from one warm pool. Assert no pod carries + two claims (the optimistic lock holds), no warm capacity leaks (guarded + release), and every claim resolves to Ready or a clean pend, never corruption. +- Throughput (uses the #15 harness): run `bench/claim --mode sustained` against a + controller scaled to N shards and confirm aggregate achieved claims/sec scales + with N until another resource (etcd write rate, forkd) becomes the wall. + +## The measurement signal that opens the gate + +Flip sharding on only when the #15 item 2 data shows ALL of: achieved claims/sec +plateaus below the arrival rate; the active leader's reconcile work-queue depth +grows without draining; and the leader pod is CPU-bound (not blocked on etcd or +forkd). If instead the wall is raw etcd write throughput (leader CPU idle, etcd +fsync latency climbing), sharding the reconciler will NOT help and the decision +returns to ADR 0009 Option C or etcd-side work; record that outcome as a new ADR. + +## Scope boundaries (YAGNI) + +- No dynamic auto-rebalancing in v1: shard-count is a deliberate operator resize. +- No mutating webhook and no per-object shard label: hash-in-predicate only, so + there is zero added per-sandbox etcd write. +- Options B and C of ADR 0009 (moving the claim/activation into the gateway) are + out of scope; they require their own ADR and a named security review. +- Step 1 (release hardening) may land independently; it is a correctness fix, not + a scaling change, and does not need the gate. + +## Threat-model delta (applies when step 2/3 land) + +Sharding introduces no new cross-tenant surface: no object is mutated, the +namespace boundary and `huskPodOwnedByPool` provenance gate are unchanged, and the +claim/activation stays in the controller. The one delta to record in +`docs/threat-model.md` when step 1 lands: the claim release is now +claim-name-guarded and optimistic-locked, closing the "stale actor blindly +un-claims a live pod" hazard that the single-writer assumption previously covered.