ci: measure the control-plane throughput ceiling on kind

[stubbi]

Thinking Path

• Mitos boots Firecracker microVMs and forks them via copy-on-write snapshots, exposed through CRDs on Kubernetes; the claim path writes a Sandbox CR per create.
• The controller + pool path is the subsystem: a claim is a Sandbox CR create, reconciled to Ready via node select and a fork, observed by a watch.
• The gap: issue [EPIC] Benchmark program: reproducible numbers or no numbers #15 item 2 asks for sustained claims/sec and a density curve. The harness (bench/claim --mode sustained) shipped and is unit-tested, but no number was ever produced, so the control-plane throughput was an unmeasured assertion.
• It needs addressing now because whether the Kubernetes control plane scales per-sandbox is the open question behind ADR 0009; a measured ceiling replaces the assertion. This serves ROADMAP issue [EPIC] Benchmark program: reproducible numbers or no numbers #15.
• This pull request runs the harness in the kind-e2e job on every CI run and records the achieved control-plane claims/sec plus per-node density as an artifact.
• The benefit is a real, reproducible control-plane ceiling produced by a live control plane, per the no-unverified-claims rule, instead of a hardcoded or hand-waved figure.

Linked Issues or Issue Description

Related: #15

What Changed

• .github/workflows/ci.yaml: after the mock control plane is Ready and the dev-default pool has a snapshot, run bench/claim --mode sustained (rate 5/s, 20s, max-concurrent 8) against it; assert only correctness (completed > 0, achieved/sec > 0) so shared-runner variance cannot flake it; upload the result JSON as the control-plane-throughput-mock artifact.
• bench/results/control-plane-throughput-method.md: the method and, explicitly, what the number is (the etcd/apiserver/reconciler ceiling on one kind node, mock engine) and is not (hardware density; the warm-husk activation path).
• BENCHMARKS.md: update the [EPIC] Benchmark program: reproducible numbers or no numbers #15 item 2 status to point at the CI datapoint and the method note.

Verification

• actionlint .github/workflows/ci.yaml: clean for the added steps (the only two warnings it reports are pre-existing, at lines far from this change).
• go build ./bench/claim/: the harness the step invokes builds.
• YAML parses (actionlint loads the whole workflow; step indentation confirmed).
• No em or en dashes in the diff.
• The number is produced on a live cluster in-job, never hardcoded; the density curve is swept with bench/sustained-claims-throughput.sh per the method note.

Risks

Low risk. The step runs after the existing "sandbox create reaches Ready" step on the same mock control plane and gates on correctness only, so it adds no new flake surface. Worst case is a genuine control-plane regression (claims stop reaching Ready under load), which is exactly what it should catch; on failure it dumps sandbox state and controller logs.

Model Used

Claude Opus 4.8 (1M context), model id claude-opus-4-8, high reasoning effort.

Checklist

• PR title is a conventional commit (ci)
• Thinking Path traces from project context to this change
• Model Used is filled in (with version and capability details)
• Tests added for behavior changes, in the same commit (TDD) (n/a: the harness aggregation is already unit-tested in internal/benchstat; this wires it into CI)
• Docs updated in the same PR (BENCHMARKS.md, method note)
• Threat-model delta included if the security surface moved (n/a: no surface change)
• Benchmark run included if the hot path was touched (this IS the benchmark wiring; it produces the number in CI)
• No em or en dashes introduced anywhere
• Secret values never logged, in errors, in condition messages, or on host paths
• No internal/instance-local references (only public #NNN)
• Every commit carries a Signed-off-by trailer

[coderabbitai]

Warning

Review limit reached

@stubbi, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 58 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6ede6d84-f92b-4a8c-9cb8-38914b823998

📥 Commits

Reviewing files that changed from the base of the PR and between 796043c and 5d812bd.

📒 Files selected for processing (3)

• .github/workflows/ci.yaml
• BENCHMARKS.md
• bench/results/control-plane-throughput-method.md

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci/control-plane-throughput-gate

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[stubbi]

First recorded datapoint from this jobs kind-e2e run (artifact control-plane-throughput-mock):

achieved_per_sec : 1.96
completed        : 42
peak_concurrent  : 8
per_node_density : { mitos-ci-worker: 42 }
window_ms        : 21408

Target arrival was 5/s; achieved was ~2/s. That gap (achieved below target) is the saturation signature the method note describes, so even this single kind node with the mock engine already shows the control-plane knee. This number is heavily shaped by the CI environment (kind etcd on slow disk, mock forkd gRPC, shared-runner CPU) and is NOT a production or hardware-density claim, per bench/results/control-plane-throughput-method.md. The reference-node sweep (issue #16) traces the real curve.