From 71bace41612e05d436b13ee29dc94f4a653b8a1f Mon Sep 17 00:00:00 2001 From: Weiming Long Date: Mon, 6 Jul 2026 19:53:47 +0800 Subject: [PATCH] Publish a measured performance report with a regeneration script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/PERFORMANCE.md carries Criterion medians for every hot-path stage — wire deserialization, dispatcher normalize with the shadow-book cross-check, WAL codec encode and append, book operations, and the read-model snapshot — stamped with the CPU, RAM, OS, toolchain, commit, and date they were measured on, plus the full 49-benchmark result table. scripts/bench-report.py regenerates it from target/criterion output and `just bench-report` wraps the run. The README benchmarks section now leads with the measured numbers instead of bare commands. Numbers replace the order-of-magnitude estimates that docs previously carried without attribution; CI continues to compile (not time) the benches, since shared runners are too noisy for statistical gating. --- CHANGELOG.md | 6 ++ README.md | 8 ++ docs/PERFORMANCE.md | 93 ++++++++++++++++++++ justfile | 6 ++ scripts/bench-report.py | 187 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 300 insertions(+) create mode 100644 docs/PERFORMANCE.md create mode 100644 scripts/bench-report.py diff --git a/CHANGELOG.md b/CHANGELOG.md index b0396afa..5f486eca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,12 @@ is practical to do so. ## [Unreleased] +- Published performance report: `docs/PERFORMANCE.md` carries measured, + machine/toolchain/commit-stamped Criterion medians for every hot-path stage + (wire deserialize, dispatcher normalize + shadow-book, WAL encode/append, + book operations, read model), regenerated with `just bench-report` + (`scripts/bench-report.py`). The README benchmarks section now leads with + the measured numbers. - Engineering-narrative documentation: a root-level `TESTING.md` mapping each failure mode to the layer that catches it (with verified counts and run commands, plus an explicit not-yet-covered section), and four new ADRs diff --git a/README.md b/README.md index b9331b99..a1f0a43c 100644 --- a/README.md +++ b/README.md @@ -231,10 +231,18 @@ bug, with real counts and run commands — lives in [TESTING.md](TESTING.md). ### Benchmarks +Measured results — hardware- and commit-stamped medians for every hot-path +stage (wire deserialize, dispatcher normalize, WAL append, book ops, read +model) — live in [docs/PERFORMANCE.md](docs/PERFORMANCE.md), regenerated with +`just bench-report`. Highlights from the committed run (Apple M3): book delta +apply 9 ns, WAL codec encode 87 ns, WAL append+flush 509 ns/record, dispatcher +normalize with shadow-book cross-check 484 ns/entry (2.1 M entries/s). + ```bash cargo bench # all benchmarks cargo bench -p pb-types # fixed-point ops + wire deserialization throughput cargo bench -p pb-book # book operations, depth iteration, mixed workloads +just bench-report # full suite + regenerate docs/PERFORMANCE.md ``` ### Property-Based Testing diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md new file mode 100644 index 00000000..135c1a29 --- /dev/null +++ b/docs/PERFORMANCE.md @@ -0,0 +1,93 @@ +# Performance + +Measured Criterion results for the hot-path operations, regenerated with +`just bench-report` (which runs `cargo bench --workspace` and rewrites this +file). Numbers are medians of Criterion's sampled iterations on the machine +below — single-machine, wall-clock measurements for order-of-magnitude +reasoning, not a controlled lab benchmark. CI compiles every benchmark on +each PR (`bench` job) but does not gate on timings: shared runners are too +noisy for statistical regression detection, so regression checks are run +locally against this file. + +## Measurement context + +- CPU: Apple M3 (8 cores), RAM: 16 GB +- OS: Darwin 25.5.0 (arm64) +- Toolchain: rustc 1.94.0 (4a4ef493e 2026-03-02), bench profile (inherits release: thin LTO, overflow-checks on) +- Commit: `e9ecd2a`, measured 2026-07-06 + +## Pipeline hot path + +| Stage | Operation | Median | Per item | Rate | +|---|---|---|---|---| +| ingest | WS `price_change` deserialize (zero-copy) | 504 ns | 504 ns | 2.0 M/s | +| ingest | WS book snapshot deserialize (10 levels) | 2.33 µs | 2.33 µs | 429 k/s | +| ingest | Dispatcher normalize + shadow-book cross-check | 483.68 µs | 484 ns | 2.1 M/s | +| durability | WAL codec encode (book delta) | 87 ns | 87 ns | 11.5 M/s | +| durability | WAL append + flush | 508.71 µs | 509 ns | 2.0 M/s | +| durability | WAL append + fdatasync every record | 403.98 ms | 4.04 ms | 248/s | +| book | Book delta apply | 9 ns | 9 ns | 107.9 M/s | +| book | Book snapshot rebuild (50 levels) | 1.10 µs | 1.10 µs | 907 k/s | +| book | Top-of-book read (best bid + ask) | 1 ns | 1 ns | 760.5 M/s | +| serving | Read-model snapshot (50 levels, depth 20) | 154 ns | 154 ns | 6.5 M/s | +| book | Mixed workload: deltas on a 20-level book | 119.68 µs | 12 ns | 83.6 M/s | + +Batch benches (dispatcher, WAL, mixed workload) time the whole batch per +iteration; the per-item column divides by the batch size. + +## All results + +| Benchmark | Median | Mean | Std dev | +|---|---|---|---| +| `1M delta applies` | 10.36 ms | 10.30 ms | 401.80 µs | +| `FixedPrice comparison` | 0 ns | 0 ns | 0 ns | +| `FixedPrice serde roundtrip` | 50 ns | 50 ns | 0 ns | +| `FixedPrice__from_f64` | 2 ns | 2 ns | 0 ns | +| `FixedPrice__try_from str` | 10 ns | 10 ns | 0 ns | +| `FixedSize__from_f64` | 2 ns | 2 ns | 0 ns | +| `L2Book__apply_delta` | 9 ns | 9 ns | 0 ns | +| `L2Book__apply_snapshot (50 levels)` | 1.10 µs | 1.10 µs | 3 ns | +| `L2Book__best_bid + best_ask` | 1 ns | 1 ns | 0 ns | +| `L2Book__mid_price` | 1 ns | 1 ns | 0 ns | +| `LiveReadModel__active_assets` | 102 ns | 102 ns | 1 ns | +| `LiveReadModel__feed_status_raw` | 60 ns | 60 ns | 0 ns | +| `LiveReadModel__is_asset_active` | 23 ns | 23 ns | 0 ns | +| `LiveReadModel__snapshot (50 levels, depth=20)` | 154 ns | 155 ns | 1 ns | +| `analytics/check_integrity` | 3 ns | 3 ns | 0 ns | +| `analytics/top_5_asks` | 26 ns | 26 ns | 0 ns | +| `analytics/top_5_bids` | 26 ns | 26 ns | 0 ns | +| `analytics/total_ask_size` | 0 ns | 0 ns | 0 ns | +| `analytics/total_bid_size` | 0 ns | 0 ns | 0 ns | +| `analytics/weighted_mid_price` | 2 ns | 2 ns | 0 ns | +| `book_depth_iteration/asks_sorted/10` | 30 ns | 30 ns | 0 ns | +| `book_depth_iteration/asks_sorted/100` | 115 ns | 115 ns | 1 ns | +| `book_depth_iteration/asks_sorted/200` | 214 ns | 215 ns | 1 ns | +| `book_depth_iteration/asks_sorted/50` | 72 ns | 72 ns | 1 ns | +| `book_depth_iteration/bids_sorted/10` | 29 ns | 29 ns | 1 ns | +| `book_depth_iteration/bids_sorted/100` | 114 ns | 115 ns | 3 ns | +| `book_depth_iteration/bids_sorted/200` | 214 ns | 215 ns | 1 ns | +| `book_depth_iteration/bids_sorted/50` | 73 ns | 72 ns | 1 ns | +| `codec__encode (book delta)` | 87 ns | 87 ns | 1 ns | +| `dispatcher/price_change normalize+shadow-book (200x5 entries)` | 483.68 µs | 487.26 µs | 11.69 µs | +| `mixed_workload/10k_deltas_on_20_level_book` | 119.68 µs | 120.22 µs | 2.64 µs | +| `snapshot_at_depth/apply_snapshot/10` | 141 ns | 141 ns | 1 ns | +| `snapshot_at_depth/apply_snapshot/100` | 2.55 µs | 2.55 µs | 20 ns | +| `snapshot_at_depth/apply_snapshot/200` | 5.70 µs | 5.71 µs | 35 ns | +| `snapshot_at_depth/apply_snapshot/50` | 1.11 µs | 1.11 µs | 3 ns | +| `spread_at_depth/mid_price/10` | 1 ns | 1 ns | 0 ns | +| `spread_at_depth/mid_price/100` | 1 ns | 1 ns | 0 ns | +| `spread_at_depth/mid_price/200` | 1 ns | 1 ns | 0 ns | +| `spread_at_depth/mid_price/50` | 1 ns | 1 ns | 0 ns | +| `spread_at_depth/spread/10` | 1 ns | 1 ns | 0 ns | +| `spread_at_depth/spread/100` | 1 ns | 1 ns | 0 ns | +| `spread_at_depth/spread/200` | 1 ns | 1 ns | 0 ns | +| `spread_at_depth/spread/50` | 1 ns | 1 ns | 0 ns | +| `wal_append/append+fdatasync-each (100 records)` | 403.98 ms | 404.91 ms | 3.27 ms | +| `wal_append/append+flush (1k records)` | 508.71 µs | 501.22 µs | 73.13 µs | +| `wire_deser/batch_100_mixed_messages` | 303.77 µs | 305.81 µs | 14.63 µs | +| `wire_deser/book_snapshot_10_levels` | 2.33 µs | 2.33 µs | 33 ns | +| `wire_deser/last_trade_price` | 259 ns | 262 ns | 5 ns | +| `wire_deser/price_change_delta` | 504 ns | 509 ns | 11 ns | + +The ClickHouse-backed cross-backend comparison bench requires a running +ClickHouse and is not part of this run. diff --git a/justfile b/justfile index f8ce6639..05daf312 100644 --- a/justfile +++ b/justfile @@ -16,6 +16,12 @@ test: bench: cargo bench +# Run the full benchmark suite and regenerate docs/PERFORMANCE.md with +# machine/toolchain/commit-stamped results +bench-report: + cargo bench --workspace + python3 scripts/bench-report.py + # Measure the durability-layer failover RTO (primary crash -> standby lease # acquire + tail recovery + resume + re-hydrate). Run on the TARGET hardware for a # realistic baseline. The full deployment RTO additionally includes container diff --git a/scripts/bench-report.py b/scripts/bench-report.py new file mode 100644 index 00000000..d85b008c --- /dev/null +++ b/scripts/bench-report.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +"""Generate docs/PERFORMANCE.md from a completed Criterion run. + +Reads target/criterion/**/new/estimates.json (produced by `cargo bench +--workspace`), stamps the result with the machine, toolchain, and commit it +was measured on, and writes a headline table plus the full result set. + +Usage: python3 scripts/bench-report.py (or: just bench-report) +""" + +import json +import platform +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +CRITERION = ROOT / "target" / "criterion" +OUT = ROOT / "docs" / "PERFORMANCE.md" + +# Headline rows: criterion id -> (label, items per iteration, pipeline stage). +# The divisor converts a per-iteration median into a per-item cost for benches +# that process a batch each iteration. +HEADLINE = { + "wire_deser/price_change_delta": ("WS `price_change` deserialize (zero-copy)", 1, "ingest"), + "wire_deser/book_snapshot_10_levels": ("WS book snapshot deserialize (10 levels)", 1, "ingest"), + "dispatcher/price_change normalize+shadow-book (200x5 entries)": ( + "Dispatcher normalize + shadow-book cross-check", + 1000, + "ingest", + ), + "codec__encode (book delta)": ("WAL codec encode (book delta)", 1, "durability"), + "wal_append/append+flush (1k records)": ("WAL append + flush", 1000, "durability"), + "wal_append/append+fdatasync-each (100 records)": ( + "WAL append + fdatasync every record", + 100, + "durability", + ), + "L2Book__apply_delta": ("Book delta apply", 1, "book"), + "L2Book__apply_snapshot (50 levels)": ("Book snapshot rebuild (50 levels)", 1, "book"), + "L2Book__best_bid + best_ask": ("Top-of-book read (best bid + ask)", 1, "book"), + "LiveReadModel__snapshot (50 levels, depth=20)": ( + "Read-model snapshot (50 levels, depth 20)", + 1, + "serving", + ), + "mixed_workload/10k_deltas_on_20_level_book": ( + "Mixed workload: deltas on a 20-level book", + 10_000, + "book", + ), +} + + +def sh(*argv: str) -> str: + return subprocess.run(argv, capture_output=True, text=True, check=False).stdout.strip() + + +def machine() -> dict: + info = { + "os": f"{platform.system()} {platform.release()} ({platform.machine()})", + "rustc": sh("rustc", "--version"), + "commit": sh("git", "-C", str(ROOT), "rev-parse", "--short", "HEAD"), + "date": datetime.now(timezone.utc).strftime("%Y-%m-%d"), + } + if platform.system() == "Darwin": + info["cpu"] = sh("sysctl", "-n", "machdep.cpu.brand_string") + info["cores"] = sh("sysctl", "-n", "hw.ncpu") + mem = sh("sysctl", "-n", "hw.memsize") + info["ram"] = f"{int(mem) // (1024**3)} GB" if mem.isdigit() else "unknown" + else: + cpu = "" + for line in Path("/proc/cpuinfo").read_text().splitlines(): + if line.startswith("model name"): + cpu = line.split(":", 1)[1].strip() + break + info["cpu"] = cpu or platform.processor() + info["cores"] = sh("nproc") + for line in Path("/proc/meminfo").read_text().splitlines(): + if line.startswith("MemTotal"): + kb = int(line.split()[1]) + info["ram"] = f"{kb // (1024**2)} GB" + break + return info + + +def fmt_ns(ns: float) -> str: + if ns < 1_000: + return f"{ns:.0f} ns" + if ns < 1_000_000: + return f"{ns / 1_000:.2f} µs" + if ns < 1_000_000_000: + return f"{ns / 1_000_000:.2f} ms" + return f"{ns / 1_000_000_000:.2f} s" + + +def fmt_rate(per_sec: float) -> str: + if per_sec >= 1_000_000: + return f"{per_sec / 1_000_000:.1f} M/s" + if per_sec >= 1_000: + return f"{per_sec / 1_000:.0f} k/s" + return f"{per_sec:.0f}/s" + + +def collect() -> dict: + results = {} + for est in sorted(CRITERION.rglob("new/estimates.json")): + bench_id = str(est.parent.parent.relative_to(CRITERION)) + data = json.loads(est.read_text()) + results[bench_id] = { + "median_ns": data["median"]["point_estimate"], + "mean_ns": data["mean"]["point_estimate"], + "stddev_ns": data["std_dev"]["point_estimate"], + } + return results + + +def main() -> int: + if not CRITERION.is_dir(): + print("no target/criterion results; run `cargo bench --workspace` first", file=sys.stderr) + return 1 + results = collect() + if not results: + print("target/criterion contains no estimates.json files", file=sys.stderr) + return 1 + m = machine() + + lines = [] + a = lines.append + a("# Performance") + a("") + a("Measured Criterion results for the hot-path operations, regenerated with") + a("`just bench-report` (which runs `cargo bench --workspace` and rewrites this") + a("file). Numbers are medians of Criterion's sampled iterations on the machine") + a("below — single-machine, wall-clock measurements for order-of-magnitude") + a("reasoning, not a controlled lab benchmark. CI compiles every benchmark on") + a("each PR (`bench` job) but does not gate on timings: shared runners are too") + a("noisy for statistical regression detection, so regression checks are run") + a("locally against this file.") + a("") + a("## Measurement context") + a("") + a(f"- CPU: {m.get('cpu', 'unknown')} ({m.get('cores', '?')} cores), RAM: {m.get('ram', 'unknown')}") + a(f"- OS: {m['os']}") + a(f"- Toolchain: {m['rustc']}, bench profile (inherits release: thin LTO, overflow-checks on)") + a(f"- Commit: `{m['commit']}`, measured {m['date']}") + a("") + a("## Pipeline hot path") + a("") + a("| Stage | Operation | Median | Per item | Rate |") + a("|---|---|---|---|---|") + for bench_id, (label, per, stage) in HEADLINE.items(): + r = results.get(bench_id) + if not r: + continue + median = r["median_ns"] + per_item = median / per + a( + f"| {stage} | {label} | {fmt_ns(median)} | {fmt_ns(per_item)} | " + f"{fmt_rate(1e9 / per_item)} |" + ) + a("") + a("Batch benches (dispatcher, WAL, mixed workload) time the whole batch per") + a("iteration; the per-item column divides by the batch size.") + a("") + a("## All results") + a("") + a("| Benchmark | Median | Mean | Std dev |") + a("|---|---|---|---|") + for bench_id, r in sorted(results.items()): + a( + f"| `{bench_id}` | {fmt_ns(r['median_ns'])} | {fmt_ns(r['mean_ns'])} | " + f"{fmt_ns(r['stddev_ns'])} |" + ) + a("") + a("The ClickHouse-backed cross-backend comparison bench requires a running") + a("ClickHouse and is not part of this run.") + a("") + + OUT.write_text("\n".join(lines)) + print(f"wrote {OUT.relative_to(ROOT)} ({len(results)} benchmarks)") + return 0 + + +if __name__ == "__main__": + sys.exit(main())