Metal: platform-aware expert-cache defaults via honest storage probe (#379)

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic

Problem

On Apple Silicon (Metal backend, unified memory, fast NVMe), colibri's stock
defaults leave a large performance win on the table. Measured campaign
(issue #379, M5 Max 128 GB):

config	tok/s
stock defaults	0.4
tuned (MTP=0 CAP_RAISE=0 --ram 90 --cap 1)	~2.0

Scope caveat, to prevent over-reading: the tuned row is a four-knob
configuration. This PR changes the defaults of exactly two of those knobs
(cap, CAP_RAISE); MTP and RAM sizing remain manual and out of scope. Stock
Metal runs with these defaults should therefore land somewhere between 0.4
and 2.0 tok/s, not at 2.0 — this PR is a partial capture of the measured
configuration, and how much of the 5x the two cache knobs alone recover is
exactly what the on-box validation run (below) will measure.

Issues #180 (M3 Ultra) and #107 (M4 Pro) corroborate the same shape on other
Apple Silicon boxes. As the maintainer noted in #379: no one maintaining this
repo has Metal hardware, and this PR was explicitly invited from the field.

Mechanism (#379)

Metal's per-buffer residency cost makes the engine's own expert LRU cache
anti-productive on fast NVMe. The OS page cache already streams cold
experts back in cheaper than Metal can re-pin them into GPU-visible buffers,
so a large expert cache just adds residency churn on top of I/O that was
already going to happen. On this class of hardware, a minimal cache (so the
engine gets out of the page cache's way) beats a large one.

Whether a volume is "fast" has to be measured, not assumed: buffered
reads lie (page-cache hits read 23-97 GB/s on the reference box; a real
F_NOCACHE read measured 14 GB/s on the same file). That's why this PR adds
an honest storage probe rather than just flipping the default unconditionally
on every Metal build.

Design

S1 -- honest storage probe (glm.c, coli_ssd_probe_raw/coli_ssd_probe_cached)
C-side, in the engine's own startup path, so it covers glm direct users and
the coli wrapper alike -- Python-only logic would miss direct users.
Mirrors compat_open_direct()/iobench.c's existing __APPLE__ branch: open
a real model shard (readdir order -- any shard is representative for a
bandwidth probe), fcntl(F_NOCACHE,1), disable readahead, random 16K-aligned
pread, single thread, ~0.35s wall-clock budget. The result is cached in
<model>/.coli_ssd (plain-text GB/s) so every startup after the first reads
a file instead of re-probing; a missing or unparsable cache file just
re-probes. The cache write is atomic (tmp+rename, the same model-dir
convention as stats_dump_q); if two engines race on a virgin dir (e.g. a
concurrent serve + run pair), both probe under mutual contention and may
cache a low reading -- the safe direction (defaults simply stay historic);
delete .coli_ssd to re-measure, and an unparsable file re-probes
automatically on the next start.

S2 -- platform-aware cache defaults (glm.c, coli, openai_server.py)
When g_metal_enabled && darwin && probe-fast:

• expert LRU cap defaults to 1/layer.
• CAP_RAISE defaults to 0 (previously 1) -- raising the cache back up
  under a generous RAM budget is the same anti-pattern the cap default exists
  to avoid.

MTP, RAM sizing, and autopin are untouched -- out of scope for this PR.

Precedence (coli_resolve_cap(), pure function, unit-tested in isolation):

tier	cap	CAP_RAISE
1 (highest)	explicit --cap N CLI	--
2	explicit CAP env	explicit CAP_RAISE env
3	platform default (Metal + darwin + fast SSD) -> 1	platform default (same condition) -> 0, independently of how cap was set -- see note
4 (lowest)	historic default: 8 via the coli wrapper ("0 = auto" sentinel), 64 for a bare ./glm with no positional argument	historic default -> 1

An explicit setting is never overridden by the probe, at any tier.

Note on the cap/CAP_RAISE coupling (deliberate): on a qualifying box, an
explicit --cap 16 does not re-enable auto-raise -- CAP_RAISE still
defaults to 0 there. Rationale: the Metal residency cost scales with cache
growth, so auto-raise is the same mechanism the minimal-cache default avoids;
a user who explicitly sizes the cache gets exactly that size. Explicit
CAP_RAISE=1 re-enables auto-raise.

S3 -- the one-line notice (glm.c, stderr, alongside the existing config banner)
Printed only when the platform cap default actually engages:

METAL: fast SSD (X.X GB/s) — page cache favored, expert cache minimal (cap 1); override with --cap

Silent when the probe ran but storage was slow -- defaults stay at their
historic values with no extra output.

S4 -- doctor/plan surfacing (doctor.py, resource_plan.py)
coli doctor and coli plan read <model>/.coli_ssd if present and display
it. Read-and-display only: neither command re-measures or guesses. A new
storage.ssd_probe doctor check reports "pass" with the cached GB/s, or
"skip" before the first Metal+darwin startup.

What's explicitly out of scope

• MTP, RAM sizing (--ram), autopin -- their defaults are untouched (and are
  the other half of the measured 5x; see the scope caveat at the top).
• Routing, CACHE_ROUTE, sampling -- nothing here can change a generated
  token. Cache sizing is a pure performance knob: it changes how often an
  expert is re-read from disk, never which expert gets computed for a given
  token.
• Non-Metal, non-darwin, or slow-storage runs: the probe never engages
  (g_metal_enabled stays 0, or the measured GB/s is under the threshold,
  default 4.0 GB/s, COLI_SSD_FAST_GBS to override), and every invocation
  resolves byte-identically to today -- including the bare ./glm fallback
  of 64.

Behavioral disclosures (small, but stated plainly)

1. --cap 0 / CAP=0 now mean "auto" on every platform, not just Metal.
   This is the sentinel that lets the wrapper say "not explicitly set."
   Previously a literal cap=0 was accepted and produced a working zero-slot,
   always-cold-load cache mode; that mode was undocumented and unused
   repo-wide (no test, tool, doc, or wrapper passes 0). Anyone who somehow
   relied on it can get the nearest supported equivalent with --cap 1.
2. coli's --cap argparse default changed from 8 to 0 (auto).
   Without this, coli always forced an explicit 8 onto the engine command
   line, which (per the precedence table) would have permanently shadowed the
   platform default for every wrapped invocation. openai_server.py's
   --cap flag and its internal Engine/serve() defaults got the same
   sentinel so programmatic callers don't silently bypass auto.
3. The benchmark harness (tools/eval_glm.py) keeps its fixed --cap 64
   deliberately -- benchmarks need a reproducible cache size, not a
   platform-dependent auto (commented in the source).

Durable vs. current-state (a review map)

Two kinds of change ship here, and they age differently:

Durable machinery -- the F_NOCACHE probe and .coli_ssd cache, the
coli_resolve_cap() precedence chain and cap sentinel, the notice
mechanism, and the doctor/plan surfacing. None of these encode an opinion
about how big the cache should be; they survive any change in residency
economics.

Current-state calibration -- the qualifying-platform values: cap 1
and CAP_RAISE 0. These encode today's measured economics (per-buffer
Metal residency cost makes a growing expert cache anti-productive, #379)
and are deliberately concentrated so that revisiting them is a two-constant
edit. "Today" is a checkable coordinate, not rhetoric: measured 2026-07 on
macOS 26.5 (M5 Max, 128 GB) against engine base caa49f7 (dev) -- the
in-code marker comments carry the same stamp. A reader far from those
coordinates (newer macOS Metal runtime, or an engine that has absorbed the
#379 residency work) should treat the constants as suspect and re-run the
#379 A/B (cap 1 vs cap 16, hit rate vs tok/s) before trusting them. Two
foreseeable developments would flip them:

1. The residency-mechanism work in Metal — growing the expert cache makes decode SLOWER #379 itself. The MTLHeap experiment
   already flips the sign of caching on the reference box (cap16 ~ +5% over
   cap1 with heap-backed slabs); a follow-up PR is planned from the same
   investigation. If it lands, the right qualifying default likely becomes a
   larger, RAM-scaled cap -- not 1.
2. Metal's move toward first-class tensor objects (MTLTensor and
   runtime-managed residency in current and upcoming macOS releases). If
   expert storage migrates there, the per-buffer registration cost -- the
   entire reason cap 1 wins today -- changes wholesale.

In both cases the edit is confined to the platform-default branch of
coli_resolve_cap() and the single CAP_RAISE default expression (both
carry a marker comment in the code); probe, precedence, notice, and
surfacing are unaffected. The COLI_SSD_FAST_GBS threshold (4.0) is
likewise a current-economics number, with the env override as escape hatch.

Testing

Author-side testing was parse-level and unit-level only -- no model was
loaded and no inference was run:

• tests/test_cap_precedence.c -- exhaustive precedence-table coverage of
  coli_resolve_cap(), including bare-invocation vs wrapper-sentinel vs
  explicit CLI/env on fast and slow platforms. Portable (no Metal/Apple
  hardware required), part of the default make test-c gate.
• tests/test_resource_plan.py / tests/test_doctor.py -- cache-file
  parsing (read_ssd_probe: missing, empty, malformed, negative, valid) and
  the doctor/plan surfacing contract.
• make glm METAL=1 and plain make glm both compile clean
  (-Wall -Wextra); full make test-c + python unittest suites pass.
• Startup-path smoke tests against a scratch directory containing only a
  dummy *.safetensors file (no config.json, so every run exits before
  model_init -- deliberately): notice printed + cap=1 on a fast volume;
  .coli_ssd written once (atomically) and reused on the next start;
  --cap 16 / CAP=4 override silently; forced-slow threshold and
  non-Metal builds stay silent at historic defaults; corrupted .coli_ssd
  re-probes; bare ./glm still resolves cap 64.

On-box validation (performed 2026-07-18): M5 Max 128 GB, macOS 26.5.2,
real GLM-5.2 int4 model, engine at this branch's HEAD (base caa49f7,
dev). Three-run block, fixed prompt,
temp 0, 32 decoded tokens each:

1. Stock defaults (no cache flags, no env): notice printed
   (METAL: fast SSD (9.7 GB/s) -- page cache favored, expert cache minimal (cap 1); override with --cap), cap resolved to 1, .coli_ssd written
   once. Decode 1.48 tok/s vs 0.4 at the old defaults -- the two cache
   knobs alone recover roughly 3.7x of the measured 5x (MTP and RAM sizing
   remain manual, as scoped above).
2. Explicit --cap 16: silent (no notice), cap honored at 16/layer,
   .coli_ssd untouched. Decode 0.86 tok/s despite the hit rate rising
   24% -> 50% -- a same-session replication of the Metal — growing the expert cache makes decode SLOWER #379 mechanism this PR
   encodes: growing the Metal-registered cache makes decode slower even as
   it hits more.
3. Restart, stock again: cached probe reused (.coli_ssd mtime unchanged,
   same 9.7 GB/s in the notice), cap=1 again, decode 1.49 tok/s
   replicating run 1; temp-0 output text identical across the two stock runs.

[JustVugg]

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.