Publish a measured performance report with a regeneration script

CHANGELOG.md

@@ -8,6 +8,12 @@ is practical to do so.
 
 ## [Unreleased]
 
+- Published performance report: `docs/PERFORMANCE.md` carries measured,
+  machine/toolchain/commit-stamped Criterion medians for every hot-path stage
+  (wire deserialize, dispatcher normalize + shadow-book, WAL encode/append,
+  book operations, read model), regenerated with `just bench-report`
+  (`scripts/bench-report.py`). The README benchmarks section now leads with
+  the measured numbers.
 - Engineering-narrative documentation: a root-level `TESTING.md` mapping each
   failure mode to the layer that catches it (with verified counts and run
   commands, plus an explicit not-yet-covered section), and four new ADRs

README.md

@@ -231,10 +231,18 @@ bug, with real counts and run commands — lives in [TESTING.md](TESTING.md).
 
 ### Benchmarks
 
+Measured results — hardware- and commit-stamped medians for every hot-path
+stage (wire deserialize, dispatcher normalize, WAL append, book ops, read
+model) — live in [docs/PERFORMANCE.md](docs/PERFORMANCE.md), regenerated with
+`just bench-report`. Highlights from the committed run (Apple M3): book delta
+apply 9 ns, WAL codec encode 87 ns, WAL append+flush 509 ns/record, dispatcher
+normalize with shadow-book cross-check 484 ns/entry (2.1 M entries/s).
+
 ```bash
 cargo bench                 # all benchmarks
 cargo bench -p pb-types     # fixed-point ops + wire deserialization throughput
 cargo bench -p pb-book      # book operations, depth iteration, mixed workloads
+just bench-report           # full suite + regenerate docs/PERFORMANCE.md
 ```
 
 ### Property-Based Testing

docs/PERFORMANCE.md

@@ -0,0 +1,93 @@
+# Performance
+
+Measured Criterion results for the hot-path operations, regenerated with
+`just bench-report` (which runs `cargo bench --workspace` and rewrites this
+file). Numbers are medians of Criterion's sampled iterations on the machine
+below — single-machine, wall-clock measurements for order-of-magnitude
+reasoning, not a controlled lab benchmark. CI compiles every benchmark on
+each PR (`bench` job) but does not gate on timings: shared runners are too
+noisy for statistical regression detection, so regression checks are run
+locally against this file.
+
+## Measurement context
+
+- CPU: Apple M3 (8 cores), RAM: 16 GB
+- OS: Darwin 25.5.0 (arm64)
+- Toolchain: rustc 1.94.0 (4a4ef493e 2026-03-02), bench profile (inherits release: thin LTO, overflow-checks on)
+- Commit: `e9ecd2a`, measured 2026-07-06
+
+## Pipeline hot path
+
+| Stage | Operation | Median | Per item | Rate |
+|---|---|---|---|---|
+| ingest | WS `price_change` deserialize (zero-copy) | 504 ns | 504 ns | 2.0 M/s |
+| ingest | WS book snapshot deserialize (10 levels) | 2.33 µs | 2.33 µs | 429 k/s |
+| ingest | Dispatcher normalize + shadow-book cross-check | 483.68 µs | 484 ns | 2.1 M/s |
+| durability | WAL codec encode (book delta) | 87 ns | 87 ns | 11.5 M/s |
+| durability | WAL append + flush | 508.71 µs | 509 ns | 2.0 M/s |
+| durability | WAL append + fdatasync every record | 403.98 ms | 4.04 ms | 248/s |
+| book | Book delta apply | 9 ns | 9 ns | 107.9 M/s |
+| book | Book snapshot rebuild (50 levels) | 1.10 µs | 1.10 µs | 907 k/s |
+| book | Top-of-book read (best bid + ask) | 1 ns | 1 ns | 760.5 M/s |
+| serving | Read-model snapshot (50 levels, depth 20) | 154 ns | 154 ns | 6.5 M/s |
+| book | Mixed workload: deltas on a 20-level book | 119.68 µs | 12 ns | 83.6 M/s |
+
+Batch benches (dispatcher, WAL, mixed workload) time the whole batch per
+iteration; the per-item column divides by the batch size.
+
+## All results
+
+| Benchmark | Median | Mean | Std dev |
+|---|---|---|---|
+| `1M delta applies` | 10.36 ms | 10.30 ms | 401.80 µs |
+| `FixedPrice comparison` | 0 ns | 0 ns | 0 ns |
+| `FixedPrice serde roundtrip` | 50 ns | 50 ns | 0 ns |
+| `FixedPrice__from_f64` | 2 ns | 2 ns | 0 ns |
+| `FixedPrice__try_from str` | 10 ns | 10 ns | 0 ns |
+| `FixedSize__from_f64` | 2 ns | 2 ns | 0 ns |
+| `L2Book__apply_delta` | 9 ns | 9 ns | 0 ns |
+| `L2Book__apply_snapshot (50 levels)` | 1.10 µs | 1.10 µs | 3 ns |
+| `L2Book__best_bid + best_ask` | 1 ns | 1 ns | 0 ns |
+| `L2Book__mid_price` | 1 ns | 1 ns | 0 ns |
+| `LiveReadModel__active_assets` | 102 ns | 102 ns | 1 ns |
+| `LiveReadModel__feed_status_raw` | 60 ns | 60 ns | 0 ns |
+| `LiveReadModel__is_asset_active` | 23 ns | 23 ns | 0 ns |
+| `LiveReadModel__snapshot (50 levels, depth=20)` | 154 ns | 155 ns | 1 ns |
+| `analytics/check_integrity` | 3 ns | 3 ns | 0 ns |
+| `analytics/top_5_asks` | 26 ns | 26 ns | 0 ns |
+| `analytics/top_5_bids` | 26 ns | 26 ns | 0 ns |
+| `analytics/total_ask_size` | 0 ns | 0 ns | 0 ns |
+| `analytics/total_bid_size` | 0 ns | 0 ns | 0 ns |
+| `analytics/weighted_mid_price` | 2 ns | 2 ns | 0 ns |
+| `book_depth_iteration/asks_sorted/10` | 30 ns | 30 ns | 0 ns |
+| `book_depth_iteration/asks_sorted/100` | 115 ns | 115 ns | 1 ns |
+| `book_depth_iteration/asks_sorted/200` | 214 ns | 215 ns | 1 ns |
+| `book_depth_iteration/asks_sorted/50` | 72 ns | 72 ns | 1 ns |
+| `book_depth_iteration/bids_sorted/10` | 29 ns | 29 ns | 1 ns |
+| `book_depth_iteration/bids_sorted/100` | 114 ns | 115 ns | 3 ns |
+| `book_depth_iteration/bids_sorted/200` | 214 ns | 215 ns | 1 ns |
+| `book_depth_iteration/bids_sorted/50` | 73 ns | 72 ns | 1 ns |
+| `codec__encode (book delta)` | 87 ns | 87 ns | 1 ns |
+| `dispatcher/price_change normalize+shadow-book (200x5 entries)` | 483.68 µs | 487.26 µs | 11.69 µs |
+| `mixed_workload/10k_deltas_on_20_level_book` | 119.68 µs | 120.22 µs | 2.64 µs |
+| `snapshot_at_depth/apply_snapshot/10` | 141 ns | 141 ns | 1 ns |
+| `snapshot_at_depth/apply_snapshot/100` | 2.55 µs | 2.55 µs | 20 ns |
+| `snapshot_at_depth/apply_snapshot/200` | 5.70 µs | 5.71 µs | 35 ns |
+| `snapshot_at_depth/apply_snapshot/50` | 1.11 µs | 1.11 µs | 3 ns |
+| `spread_at_depth/mid_price/10` | 1 ns | 1 ns | 0 ns |
+| `spread_at_depth/mid_price/100` | 1 ns | 1 ns | 0 ns |
+| `spread_at_depth/mid_price/200` | 1 ns | 1 ns | 0 ns |
+| `spread_at_depth/mid_price/50` | 1 ns | 1 ns | 0 ns |
+| `spread_at_depth/spread/10` | 1 ns | 1 ns | 0 ns |
+| `spread_at_depth/spread/100` | 1 ns | 1 ns | 0 ns |
+| `spread_at_depth/spread/200` | 1 ns | 1 ns | 0 ns |
+| `spread_at_depth/spread/50` | 1 ns | 1 ns | 0 ns |
+| `wal_append/append+fdatasync-each (100 records)` | 403.98 ms | 404.91 ms | 3.27 ms |
+| `wal_append/append+flush (1k records)` | 508.71 µs | 501.22 µs | 73.13 µs |
+| `wire_deser/batch_100_mixed_messages` | 303.77 µs | 305.81 µs | 14.63 µs |
+| `wire_deser/book_snapshot_10_levels` | 2.33 µs | 2.33 µs | 33 ns |
+| `wire_deser/last_trade_price` | 259 ns | 262 ns | 5 ns |
+| `wire_deser/price_change_delta` | 504 ns | 509 ns | 11 ns |
+
+The ClickHouse-backed cross-backend comparison bench requires a running
+ClickHouse and is not part of this run.

justfile

@@ -16,6 +16,12 @@ test:
 bench:
     cargo bench
 
+# Run the full benchmark suite and regenerate docs/PERFORMANCE.md with
+# machine/toolchain/commit-stamped results
+bench-report:
+    cargo bench --workspace
+    python3 scripts/bench-report.py
+
 # Measure the durability-layer failover RTO (primary crash -> standby lease
 # acquire + tail recovery + resume + re-hydrate). Run on the TARGET hardware for a
 # realistic baseline. The full deployment RTO additionally includes container

scripts/bench-report.py

@@ -0,0 +1,187 @@
+#!/usr/bin/env python3
+"""Generate docs/PERFORMANCE.md from a completed Criterion run.
+
+Reads target/criterion/**/new/estimates.json (produced by `cargo bench
+--workspace`), stamps the result with the machine, toolchain, and commit it
+was measured on, and writes a headline table plus the full result set.
+
+Usage: python3 scripts/bench-report.py   (or: just bench-report)
+"""
+
+import json
+import platform
+import subprocess
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+CRITERION = ROOT / "target" / "criterion"
+OUT = ROOT / "docs" / "PERFORMANCE.md"
+
+# Headline rows: criterion id -> (label, items per iteration, pipeline stage).
+# The divisor converts a per-iteration median into a per-item cost for benches
+# that process a batch each iteration.
+HEADLINE = {
+    "wire_deser/price_change_delta": ("WS `price_change` deserialize (zero-copy)", 1, "ingest"),
+    "wire_deser/book_snapshot_10_levels": ("WS book snapshot deserialize (10 levels)", 1, "ingest"),
+    "dispatcher/price_change normalize+shadow-book (200x5 entries)": (
+        "Dispatcher normalize + shadow-book cross-check",
+        1000,
+        "ingest",
+    ),
+    "codec__encode (book delta)": ("WAL codec encode (book delta)", 1, "durability"),
+    "wal_append/append+flush (1k records)": ("WAL append + flush", 1000, "durability"),
+    "wal_append/append+fdatasync-each (100 records)": (
+        "WAL append + fdatasync every record",
+        100,
+        "durability",
+    ),
+    "L2Book__apply_delta": ("Book delta apply", 1, "book"),
+    "L2Book__apply_snapshot (50 levels)": ("Book snapshot rebuild (50 levels)", 1, "book"),
+    "L2Book__best_bid + best_ask": ("Top-of-book read (best bid + ask)", 1, "book"),
+    "LiveReadModel__snapshot (50 levels, depth=20)": (
+        "Read-model snapshot (50 levels, depth 20)",
+        1,
+        "serving",
+    ),
+    "mixed_workload/10k_deltas_on_20_level_book": (
+        "Mixed workload: deltas on a 20-level book",
+        10_000,
+        "book",
+    ),
+}
+
+
+def sh(*argv: str) -> str:
+    return subprocess.run(argv, capture_output=True, text=True, check=False).stdout.strip()
+
+
+def machine() -> dict:
+    info = {
+        "os": f"{platform.system()} {platform.release()} ({platform.machine()})",
+        "rustc": sh("rustc", "--version"),
+        "commit": sh("git", "-C", str(ROOT), "rev-parse", "--short", "HEAD"),
+        "date": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
+    }
+    if platform.system() == "Darwin":
+        info["cpu"] = sh("sysctl", "-n", "machdep.cpu.brand_string")
+        info["cores"] = sh("sysctl", "-n", "hw.ncpu")
+        mem = sh("sysctl", "-n", "hw.memsize")
+        info["ram"] = f"{int(mem) // (1024**3)} GB" if mem.isdigit() else "unknown"
+    else:
+        cpu = ""
+        for line in Path("/proc/cpuinfo").read_text().splitlines():
+            if line.startswith("model name"):
+                cpu = line.split(":", 1)[1].strip()
+                break
+        info["cpu"] = cpu or platform.processor()
+        info["cores"] = sh("nproc")
+        for line in Path("/proc/meminfo").read_text().splitlines():
+            if line.startswith("MemTotal"):
+                kb = int(line.split()[1])
+                info["ram"] = f"{kb // (1024**2)} GB"
+                break
+    return info
+
+
+def fmt_ns(ns: float) -> str:
+    if ns < 1_000:
+        return f"{ns:.0f} ns"
+    if ns < 1_000_000:
+        return f"{ns / 1_000:.2f} µs"
+    if ns < 1_000_000_000:
+        return f"{ns / 1_000_000:.2f} ms"
+    return f"{ns / 1_000_000_000:.2f} s"
+
+
+def fmt_rate(per_sec: float) -> str:
+    if per_sec >= 1_000_000:
+        return f"{per_sec / 1_000_000:.1f} M/s"
+    if per_sec >= 1_000:
+        return f"{per_sec / 1_000:.0f} k/s"
+    return f"{per_sec:.0f}/s"
+
+
+def collect() -> dict:
+    results = {}
+    for est in sorted(CRITERION.rglob("new/estimates.json")):
+        bench_id = str(est.parent.parent.relative_to(CRITERION))
+        data = json.loads(est.read_text())
+        results[bench_id] = {
+            "median_ns": data["median"]["point_estimate"],
+            "mean_ns": data["mean"]["point_estimate"],
+            "stddev_ns": data["std_dev"]["point_estimate"],
+        }
+    return results
+
+
+def main() -> int:
+    if not CRITERION.is_dir():
+        print("no target/criterion results; run `cargo bench --workspace` first", file=sys.stderr)
+        return 1
+    results = collect()
+    if not results:
+        print("target/criterion contains no estimates.json files", file=sys.stderr)
+        return 1
+    m = machine()
+
+    lines = []
+    a = lines.append
+    a("# Performance")
+    a("")
+    a("Measured Criterion results for the hot-path operations, regenerated with")
+    a("`just bench-report` (which runs `cargo bench --workspace` and rewrites this")
+    a("file). Numbers are medians of Criterion's sampled iterations on the machine")
+    a("below — single-machine, wall-clock measurements for order-of-magnitude")
+    a("reasoning, not a controlled lab benchmark. CI compiles every benchmark on")
+    a("each PR (`bench` job) but does not gate on timings: shared runners are too")
+    a("noisy for statistical regression detection, so regression checks are run")
+    a("locally against this file.")
+    a("")
+    a("## Measurement context")
+    a("")
+    a(f"- CPU: {m.get('cpu', 'unknown')} ({m.get('cores', '?')} cores), RAM: {m.get('ram', 'unknown')}")
+    a(f"- OS: {m['os']}")
+    a(f"- Toolchain: {m['rustc']}, bench profile (inherits release: thin LTO, overflow-checks on)")
+    a(f"- Commit: `{m['commit']}`, measured {m['date']}")
+    a("")
+    a("## Pipeline hot path")
+    a("")
+    a("| Stage | Operation | Median | Per item | Rate |")
+    a("|---|---|---|---|---|")
+    for bench_id, (label, per, stage) in HEADLINE.items():
+        r = results.get(bench_id)
+        if not r:
+            continue
+        median = r["median_ns"]
+        per_item = median / per
+        a(
+            f"| {stage} | {label} | {fmt_ns(median)} | {fmt_ns(per_item)} | "
+            f"{fmt_rate(1e9 / per_item)} |"
+        )
+    a("")
+    a("Batch benches (dispatcher, WAL, mixed workload) time the whole batch per")
+    a("iteration; the per-item column divides by the batch size.")
+    a("")
+    a("## All results")
+    a("")
+    a("| Benchmark | Median | Mean | Std dev |")
+    a("|---|---|---|---|")
+    for bench_id, r in sorted(results.items()):
+        a(
+            f"| `{bench_id}` | {fmt_ns(r['median_ns'])} | {fmt_ns(r['mean_ns'])} | "
+            f"{fmt_ns(r['stddev_ns'])} |"
+        )
+    a("")
+    a("The ClickHouse-backed cross-backend comparison bench requires a running")
+    a("ClickHouse and is not part of this run.")
+    a("")
+
+    OUT.write_text("\n".join(lines))
+    print(f"wrote {OUT.relative_to(ROOT)} ({len(results)} benchmarks)")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())