Add bencher_replica: in-process SQLite replication to replace Litestream

[epompeii]

Why

Litestream 0.5.13 blocked all Bencher Cloud API writes for ~5.5 minutes on 2026-07-10: when its verify step decides a full re-snapshot is needed, it copies the entire 4.35 GB database into a local LTX file while holding the SQLite write lock (via its _litestream_lock table), and no configuration knob reaches that code path. The same stall recurs on process restarts, and LTX compaction churns whole-database S3 transfers several times a day. Upstream 0.5.14 does not fix it.

bencher_replica is an in-process replacement. Being in-process is the structural fix: the app funnels writes through a single writer mutex, so the replicator coordinates checkpoints with the app's own write scheduling instead of fighting a separate process for locks. The prime invariant (I5): the SQLite write lock is only ever held for O(WAL-tail) work, never O(database). All six governing invariants are documented in plus/bencher_replica/src/lib.rs.

What

• WAL parser with full salt and cumulative checksum-chain verification, cross-validated in both directions against SQLite itself
• Storage: local filesystem XOR S3-compatible (endpoint override for R2/MinIO) behind one contract, plus a scripted fault-injection wrapper for tests
• Sync engine: step-driven core; checkpoints run PASSIVE while the replicator holds BEGIN IMMEDIATE, closing the ship-vs-checkpoint race without ever needing blocking RESTART/TRUNCATE checkpoints
• Snapshots: generation-based, sourced from a single-step SQLite online backup into a scratch file (transactionally consistent under concurrent checkpoints), throttled zstd multipart upload, snapshot.json as the atomic commit marker with a mandatory-replay boundary offset
• Restore: latest-only, in the same startup handshake slot Litestream used; every epoch WAL is chain-pre-validated before application and checkpoint consumption is verified frame-for-frame
• Verification: restore-and-compare at a pinned position (default daily, 0 disables); the backstop for externally-caused divergence
• Shadow mode: with both plus.litestream and plus.replica configured, Litestream keeps checkpoint ownership and restore precedence during the burn-in
• Fix: standalone sweep connections (stats, credit grants) now disable wal_autocheckpoint when replication is configured; previously the credit sweep could checkpoint and restart the WAL behind Litestream's back
• Observability: Replica* otel counters and a critical-section histogram; JsonLitestream.metrics_port plus [[metrics]] in the Fly configs so litestream_* Prometheus metrics are scraped during the shadow period

Testing

• 275 crate tests: WAL parser fixtures (real and synthetic, both checksum byte orders, golden cross-version canary), three-backend storage contract suite, 8 fault-injection scenarios, 6 crash-recovery kill points, 8 seeded 200-op randomized equivalence workloads, ignored soak and live-S3 tiers
• 4 server-level integration tests: real API writes replicate and restore to a logically equivalent database; reboot-with-missing-DB auto-restores; shadow coexistence
• An adversarial multi-agent review confirmed 18 findings; all are fixed with regression tests. Highlights: salt-match resume now proves CONTENT against the replica tip (post-power-loss rewind forks), the meta-verified resume path requires salt1 continuity (buried WAL cycles), restore hard-fails instead of soft-stopping when local errors could leave torn state, and shutdown checkpoints after the final ship so SQLite's checkpoint-on-close WAL deletion does not force a full snapshot on every deploy
• Full gates: cargo nextest run --all-features (2047 passed), cargo test --doc, cargo clippy --no-deps --all-targets --all-features -- -Dwarnings, cargo check --no-default-features, cargo gen-types

Deployment plan

1. Shadow burn-in: add plus.replica (S3 target, its own bucket/prefix) alongside the existing plus.litestream config; watch replica.verify.pass and replica.divergence metrics
2. Run the ignored live-S3 tier against a real bucket before the shadow deploy: cargo nextest run -p bencher_replica --features plus,testing --run-ignored ignored-only with BENCHER_REPLICA_TEST_S3_* env vars
3. Cutover: remove the plus.litestream section (forces one clean generation); the Litestream removal checklist (binary, Dockerfile stage, config types) follows after the burn-in

Notes for review

• One new lint suppression needs sign-off per CLAUDE.md: #[expect(clippy::too_many_lines)] on ApiCounter::description() in bencher_otel (matches existing precedent for exhaustive metadata matches)
• cargo deny check fails on two PRE-EXISTING advisories via bencher_plot's tree (ttf-parser unmaintained, crossbeam-epoch RUSTSEC-2026-0204), unrelated to this change
• Documented residual limitation: an external writer that buries a WAL cycle AND deletes the WAL file while the server is down is undetectable at resume; the daily verification is the backstop

[github-actions]

🤖 Claude Code Review

PR: #929
Base: devel
Head: u/ep/bencher-replica
Commit: 4c2c1c5e6f6b175029d259641891f307bc877638

---

Another leftover timer — no action needed. The review is complete.

---

_{Model: claude-opus-4-8}

[github-actions]

Bencher Report

Branch	u/ep/bencher-replica
Testbed	intel-v1

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	4.63 µs (+0.11%) Baseline: 4.63 µs	4.90 µs (94.55%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	4.49 µs (-0.16%) Baseline: 4.50 µs	4.72 µs (95.19%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	26.20 µs (+2.53%) Baseline: 25.55 µs	26.68 µs (98.19%)
Adapter::Rust	📈 view plot 🚷 view threshold	3.53 µs (+1.12%) Baseline: 3.49 µs	3.60 µs (97.90%)
Adapter::RustBench	📈 view plot 🚷 view threshold	3.51 µs (+0.88%) Baseline: 3.48 µs	3.59 µs (97.79%)

🐰 View full continuous benchmarking report in Bencher