feat: nightly whole-process RSS capacity tier (large, isolated) — follow-up to the benchmark suite

[FBumann]

Follow-up to the benchmark suite. The per-PR suite tracks per-phase time + logical memory (CodSpeed) and the end-to-end pipeline (build→export logical peak). What it deliberately does not track is physical RSS — the "will this OOM / fit in N GB" number. This issue captures the design for adding it as a separate nightly tier, decided after prototyping it inside the suite and (correctly) ripping it back out.

Why a separate tier, not the per-PR suite

• Coupling: it needs an isolated-RSS tool (pytest-benchmem / getrusage); the per-PR suite must stay coupled to nothing but pytest-benchmark + CodSpeed.
• Coarse + slow: RSS is ~2% CV and needs a fresh process per measurement — wrong for the fast path.
• Floor-dominated at PR sizes: RSS only carries signal when the workload dwarfs the ~150 MB interpreter/import floor. At the per-PR sweep sizes it's mostly floor.
• It can't be derived from CodSpeed's logical memory — no fixed ratio (measured: pypsa_scigrid rss≫peak, build-resident; kvl_cycles peak≫rss, transient). So it must be measured directly.

Design principles

1. Nightly, separate job (not per-PR), outside benchmarks/.
2. Large sizes only — pushed well above the per-PR sweep, capped under the runner's RAM.
3. Whole-job, isolated — build→export in one fresh process; read getrusage ru_maxrss (or cgroup memory.peak for solver-inclusive). A phase's RSS can't be isolated — capacity is whole-job.
4. Generous gate on a stable stat — min-of-N, thresholds ≥5–10%.
5. Picklable/rebuild-in-child recipes (partial(build_and_export, spec, n)), so the child rebuilds cold rather than receiving a pre-built model.

Two flavors (pick by goal)

• A — OOM safety net (simplest): nightly CI runs a few large cases under a hard cap and asserts they survive (systemd-run --scope -p MemoryMax=… / /usr/bin/time -v + threshold). Binary, deployment-truthful, zero tracking infra.
• B — RSS trend KPI (fits a sparsity fork): standalone getrusage worker over large whole-job cases → emit Bencher Metric Format → bencher run. Hosted chart of RSS trending down + threshold alerts. (Bencher is metric-agnostic; the pytest adapter doesn't capture RSS, so use the json/BMF adapter.)

Prototype reference

The capacity.py + capacity_rss.py pattern was built and removed during the suite work — the right idea in the wrong place (kept trying to fold it into the per-PR suite). Resurrect it as a standalone nightly job: top-level picklable recipes, a few large cases (expression_arithmetic@1500 ≈ 2.3 GB, sparse_network@1000 ≈ 1.9 GB, pypsa_scigrid@50 ≈ 1.3 GB — the heaviest, real model), min-of-3, ~50 s total.

Motivating findings (from the suite work)

• pypsa_scigrid is the memory ceiling (1.3 GB rss, build-dominated) — a single-spec set would miss it; the capacity cases must be model-diverse.
• Logical and physical diverge with no fixed ratio — RSS is its own metric.
• The investigation also surfaced a real slice_size 3.4× write-memory lever in the LP writer (separate optimization).

Open decisions

• A (cap-assert) vs B (Bencher trend) — A first, graduate to B if the trend gets watched.
• Which large cases + sizes (under the target runner's RAM).
• Solver-inclusive (cgroup memory.peak, Linux-only) — only if gating solver memory becomes a need.

(AI-assisted; the numbers are real measurements from the suite work.)