perf: warm-on-write read caching + leaf-split to fix reindex-churn read regression

[bplatz]

Problem

Under continuous writes with a low reindex_min_bytes, frequent reindexing was
hurting read latency instead of helping — the opposite of the expected
"binary index ≫ novelty query" behavior. A sweep showed the default (100 KB)
churn regime collapsing product-detail read latency ~30× versus a 2 MB setting.

Root cause was not cache invalidation (the shared cache is a single global Arc
and CID-keyed, so unchanged warmth survives a swap). It was that every reindex
swap left the touched artifacts cold — the incremental build decoded the leaves
it rewrote but never seeded the shared read cache — compounded by local-read I/O
amplification and unbounded trailing-leaf growth.

Changes

• Warm-on-write: the incremental CoW build now seeds the shared LeafletCache
  with the leaves it just wrote (decoded under ColumnSet::ALL) and warms the
  reverse-dict leaves, so the first read after a swap hits warm instead of
  cold-decoding. Wired through a late-bound WarmCacheSource at a single
  chokepoint (start_background_indexing_dyn) so every build path is covered.
• Superset cache fallback: a narrow (projection-keyed) read is served from a
  cached ALL batch via ColumnBatch::project_to (invariant: cached ⊇ requested),
  so warming does not duplicate column data across cache entries.
• Bounded incremental leaves: oversized touched leaves are split during the
  incremental build — gap-free, with first_key → next-first-key boundaries and
  ±∞ sentinels at the ends — fixing unbounded growth of the trailing leaf.
• Local read path: mmap local leaf reads and cache the decoded directory.
  Previously a local read did std::fs::read on the whole leaf to serve one of
  its leaflets and re-decoded the directory, bypassing the CID-keyed LeafDir
  cache.
• Within-leaflet seek: binary-search the bound leading key within a leaflet
  scan instead of scanning the leaflet linearly to find the subject's rows.

Bench

Adds reindex_swap_read_profile, a deterministic read-after-reindex-swap harness
that isolates the per-swap cold cost as a single number (not inferred from QMpH),
with first-touch and cold-subject controls.

Validation

Existing-data reads after a churn swap drop from ~3 ms to ~0.2–0.3 ms (~10×) in
the read-after-swap microbench, with cold-miss cache insertions driven toward
zero. Query correctness is unchanged (query and api group suites pass).

Scope note

A follow-on experiment — warming the per-generation query caches (overlay
translation + stats view) at index apply — was prototyped and dropped: it is flat
on BSBM update (the between-swap overlay is small and the query:swap ratio is
high, so the per-swap translation cost amortizes to noise), and the remaining
default-regime cost is the co-located background build competing for CPU, which is
a separate lever tracked for later.

[aaj3f]

This feels like a good find/catch!

[aaj3f]

warm_leaf_into_cache runs a synchronous full zstd decode of every leaflet of every rewritten leaf (load_leaflet_columns at line 3850) directly on the async uploader task. upload_one_leaf_blob is awaited inside the async move uploader loop in run_update_branch (line ~340), i.e. on a Tokio worker thread — not the spawn_blocking CoW task (line 308). In the co-located deployment (the only case where warm_cache is Some) that runtime also serves queries, so a churn build can stall query-serving worker threads on decompression — partially working against the latency win this PR chases. (Verified: impact is bounded — the uploader is a single sequential future, so at most one runtime worker is intermittently occupied by a decode (≈1/num_cpus of capacity). Real anti-pattern and worth fixing, but a recommended cleanup, not a hard merge blocker.) Move the decode off the async path:

// in upload_one_leaf_blob, replace the inline call:
if let Some(cache) = warm_cache {
    let cache = cache.clone();            // Arc<LeafletCache>
    let leaf_cid = info.leaf_cid.clone();
    let leaf_bytes = info.leaf_bytes.clone(); // or Arc the bytes to avoid the copy
    tokio::task::spawn_blocking(move || {
        warm_leaf_into_cache(&cache, &leaf_cid, &leaf_bytes);
    });
}

(Spawn-and-forget keeps the upload off the critical path; or thread the warm into the existing spawn_blocking CoW where the bytes already exist. At minimum, wrap in spawn_blocking so a large leaf's decompression cannot block the async reactor.)

[aaj3f]

Warming inserts a ColumnSet::ALL batch per leaflet. ALL is wider than any read projection (for_scan never requests t unless time-travel), so warm-on-write pushes more bytes into the shared, TinyLFU-bounded cache than the reads it serves would — under heavy churn this can evict hot narrow entries that queries actually touch. This is the deliberate "one batch serves every projection" tradeoff, but consider (a) warming only when the leaf is small enough to fit a per-build warm budget, or (b) skipping the t column (warm ALL & !T) since reads rarely need it. Not a blocker; worth a budget knob.

// consider a bounded warm: skip leaflets whose ALL-decode would exceed a
// per-build byte budget, so a giant rewritten leaf can't evict the working set.