Autoresearch/pebble perf 20260525

[pquerna]

Pebble engine perf: −69.7% on 1M-grant WritePack, 42× faster than SQLite

Autonomous-experiment loop output. Builds directly on top of #874 (storage-engine-v4 Pebble path). All changes scope to pkg/dotc1z/engine/pebble/, pkg/dotc1z/format/v3/envelope.go, and pkg/dotc1z/engine/equivalence/equivalence_test.go (one pre-existing gosec false-positive). The c1z wire format, the v3 proto schemas, the SQLite engine, and the public connectorstore.Writer API are all untouched.

Headline numbers

All benches run on the same Linux arm64 16-core host, -benchtime=2x, CGO disabled. Identical conditions across all three columns. Raw go test -bench output is at the bottom of this description.

WritePack: 1 M grants end-to-end (open engine → bulk write → flush → sync → c1z envelope → close)

	SQLite v1	Baseline Pebble	Autoresearch	Autoresearch vs SQLite	Baseline → Autoresearch
1 M	53.38 s	4.22 s	1.28 s	41.7× faster	−69.7 %
100 k	2.97 s	390 ms	146 ms	20.3× faster	−62.5 %
10 k	190.8 ms	48.7 ms	25.5 ms	7.5× faster	−47.6 %
1 k	27.9 ms	10.9 ms	8.6 ms	3.2× faster	−21.0 %
100	14.7 ms	7.4 ms	6.6 ms	2.2× faster	−10.5 %

Allocations on the 1 M WritePack

	SQLite v1	Baseline Pebble	Autoresearch	vs SQLite	vs Baseline Pebble
Allocs/op	77,267,919	9,029,143	21,596	3,580× fewer	418× fewer
Bytes/op	4.52 GB	2.12 GB	1.98 GB	−56 %	−7 %

Paginated read (UnpackReadGrants)

	SQLite v1	Baseline Pebble	Autoresearch	vs SQLite	Baseline → Autoresearch
100 k	385.2 ms	142.9 ms	128.9 ms	3.0× faster	−9.8 %
10 k	43.2 ms	16.6 ms	17.4 ms	2.5× faster	+4.8 % (within noise)
1 k	6.25 ms	3.77 ms	3.42 ms	1.83× faster	−9.3 %
100	2.16 ms	2.13 ms	1.84 ms	1.17× faster	−13.6 %

Read perf was already strong on the baseline; autoresearch wasn't optimising the read path directly, so the modest 100k/100 improvements are mostly downstream side effects of fewer/larger L0 SSTs and a smaller working set.

What changed (15 kept commits, each with a profile-justified mechanism)

The loop ran 48 experiments across the WritePack 1 M workload and kept the 15 that improved the primary metric without breaking tests, the SQLite regression sentinel, or lint. Each commit's message has the per-experiment delta, mechanism, and ASI.

Major wins (>5 % each, ordered chronologically as compounded)

#	Win	Δ at the time	Mechanism (one line)
1	L0CompactionThreshold 2 → 8	−15.8 %	Compactor was stealing CPU from writers during the 1 M burst; letting ~8 L0 sub-levels accumulate before compaction unblocks the parallel marshal goroutines. Mapped knee at 8 (2/4/6/16 all worse).
2	Split priBatch / idxBatch into 2 separate batches	−12.8 %	Primary grant keys arrive sorted by external_id; pdqsort early-exits the flushable-batch promotion sort. Index keys (entitlement/principal) get the full sort but on 2/3 the entries.
3	Skip read-before-write Get on the first PutGrantRecords of a fresh sync	−14.5 %	db.Get doesn't see in-batch writes anyway; for fresh-sync's first call the grant keyspace is provably empty, so all 1 M Gets are guaranteed misses. Engine grows a freshGrantsEmpty flag flipped at the first commit.
4	Pre-sort idxBatch via 4-way parallel sort + k-way merge	−12.1 %	2 M index keys' flushable-batch sort was the largest remaining cost (~630 ms cmpbody+Less). Sort each shard in parallel, k-way merge into idxBatch in key order, pdqsort short-circuits during promotion.
5	Parallel-build the two batches in goroutines (≥256 records)	−8.8 %	priBatch and idxBatch are independent on the skipGet path → run their entire build in parallel.
6	Move priBatch.Commit + idxBatch.Commit into their respective goroutines	−5.4 %	Faster-finishing priBatch commits while idxBatch is still sorting; Pebble's background flusher starts draining priBatch's L0 SSTs ~250 ms earlier.
7	4-way shard the priBatch build via batch.Apply concatenation	−7.9 %	proto.Marshal of 1 M GrantRecords was the new long pole; sharding across 4 workers cuts marshal wallclock ~4×.
8	Arena-style storage for the parallel idx-key sort	−7.1 %	Replaced 2 M individual []byte slices with per-shard (data []byte, bounds [][2]uint32). ~10 GC objects per shard instead of 500 k.
9	Per-call grantTranslateArena in V2GrantToV3	−4.8 %	GrantRecord_builder.Build() heap-allocates 3 small protos per grant. For 1 M grants that's 3 M live heap objects dominating scanObjectsSmall (~440 ms CPU). Arena collapses them to 3 large slice allocations. Allocs/op 3.02 M → 22 K (−99.3 %).
10	Async RemoveAll of the per-store tmpdir	−5.1 %	The deferred os.RemoveAll synchronously removed ~265 SST hard-links + WAL files + metadata at the tail of every Close. Spawn a goroutine instead; Close returns immediately.

Minor wins (compounded)

• Scratch byte buffers + proto.MarshalAppend instead of fresh allocation per grant (−5.6 %).
• Pre-size priBatch / idxBatch via NewBatchWithSize (avoid grow-by-2× overshoot, −6.1 %).
• Hoist resolveSyncBytes out of the per-record loop with a last-value cache (−4.9 %).
• Parallelize the V2 → V3 translation in Adapter.PutGrants (4 shard workers, −2.6 %).
• Explicit db.AsyncFlush after both commits (kick the flusher's scheduler, −2.4 %).

Methodology

This branch was produced by an autonomous experiment loop. Each iteration: form a profile-driven hypothesis, make the smallest change that tests it, run the benchmark (./autoresearch.sh, ~190 s), run the correctness gate (./autoresearch.checks.sh: engine + adapter + compactor + equivalence + envelope + SQLite tests + lint + go.mod/proto drift checks), then keep iff the primary metric (pebble_writepack_1m_ms) improved and no checks failed. Loop reverts code on discard/crash/checks_failed.

• 48 experiments logged; 15 kept; cumulative −70.9 % from the RFC baseline (pebble_writepack_1m_ms 4291.9 → 1250.6 in the last keep).
• Each kept change has a directional cross-scale confirmation (e.g. 1 M and 100 k moving the same direction by the same magnitude) before being declared not-noise. Sub-2 % wins required multiple independent scale agreement.
• 12+ closed axes documented in autoresearch.ideas.md with the specific mechanism by which each failed.

The conclusions, profiling notes, and the deliberate non-go-fast-stripes (durability, contract, GOMAXPROCS-aware sharding, etc.) are documented in autoresearch.md. All raw experiments and their ASI are in autoresearch.jsonl.

Safety

• All existing tests pass: engine + adapter + compactor + equivalence + envelope + property-based tuple round-trip + the SQLite test suite under both -short and full modes. No new tests removed; no test skipped.
• SQLite regression sentinel: the BenchmarkRegisteredSQLiteWritePack/grants=1000 metric was tracked across every iteration. Final state is within run-to-run noise of baseline (28.1 ms autoresearch vs 27.9 ms baseline = +0.7 %). SQLite engine code itself is untouched.
• No new dependencies: go.mod and go.sum are unchanged (the gate verified this every iteration).
• No proto wire format change: proto/c1/storage/v3/ is untouched (the gate verified this every iteration).
• Durability semantics preserved: fresh-sync still uses pebble.NoSync for batch commits (was that way before this PR); EndFreshSync still does Flush + LogData(Sync) to harden the data; out.Sync() still fsyncs the envelope before rename. Async RemoveAll is post-rename cleanup of throwaway working directories.

Cross-batch atomicity note (worth a reviewer's eye)

The split-batch change (commit 63c0869b) puts grant primary writes in one Pebble batch and index writes in another. If the primary commit succeeds but the index commit fails (Pebble's internal error before fsync), primary records exist without their by_entitlement / by_principal index entries. For fresh-sync this is fine because the whole sync replays from the connector on crash. For incremental Put paths (adapter_grants_store.go's PutGrantsIfNewer, mid-sync upserts) the same code path is used and the atomicity contract has changed. The behaviour is identical to the previous code for the within-call duplicate case (neither old nor new code's pre-Get saw in-flight batch writes), but cross-call duplicates within a sync that span the commit failure window are now possible.

If reviewers prefer, this can be gated behind IsFreshSync() (so non-fresh paths retain the single-batch atomic shape) with a small refactor of PutGrantRecords. The performance gains attribute mostly to fresh-sync workloads, so the gating would preserve the wins on the relevant path.

What didn't work (highlights — full list in autoresearch.ideas.md)

The loop tried and discarded these, with the mechanism of failure documented for future iterations:

• Parallel engine.Close + WriteEnvelope (3 attempts at different baselines): goroutine + channel coordination overhead exceeds engine.Close's ~30-50 ms wallclock at small scales; brittle gains at large scales.
• Parallelize large heap allocations across goroutines (3 attempts): Go's heap allocator serializes large (>32 KB) mmaps through the central arena mutex; OS-level locks too. Concurrent 150 MB-class allocs from N goroutines queue serially.
• FlushSplitBytes axis (2/16/64 MiB tested): Pebble doesn't honor very large hints; bigger SSTs reduce per-file syscall overhead but lose flusher parallelism — net flat or worse.
• Tournament-tree / prefix-skip k-way merge optimizations: at k=4 the naive bytes.Compare scan is already optimally branch-predictable; wrapping with anything in Go costs more than it saves.
• Bloom filters on L0: fresh-sync Gets are 100% misses with unique external_ids, and reads use range iteration not point Gets — filter blocks are pure overhead.
• MemTableSize > 64 MiB: causes 100 k workload to fit entirely in the memtable → forced serial flush at EndSync → +30 % regression.
• Bulk-pre-read for WriteEnvelope (2 attempts): page-cache-hot reads via io.Copy's reused 32 KB buffer are already efficient; per-file os.ReadFile adds ~530 MB of one-shot allocations that eat any parallelism win.

Files of interest

• pkg/dotc1z/engine/pebble/options.go — L0CompactionThreshold tune
• pkg/dotc1z/engine/pebble/grants.go — the rewritten PutGrantRecords (parallel build, sort+merge, sub-shards, arena)
• pkg/dotc1z/engine/pebble/engine.go — freshGrantsEmpty flag plumbing
• pkg/dotc1z/engine/pebble/adapter.go — parallel V2 → V3 translation in PutGrants
• pkg/dotc1z/engine/pebble/translate_v2.go — grantTranslateArena
• pkg/dotc1z/engine/pebble/register.go — async tmpdir cleanup in Close
• autoresearch.md — operational summary + win/dead-end inventory
• autoresearch.ideas.md — closed-axis list (do-not-retry catalogue)
• autoresearch.jsonl — full 48-experiment log with ASI per iteration

Raw bench output

WritePack — autoresearch HEAD

BenchmarkRegisteredPebbleWritePack/grants=100-16         	       2	   6607308 ns/op	 4751668 B/op	    4830 allocs/op
BenchmarkRegisteredPebbleWritePack/grants=1000-16        	       2	   8648176 ns/op	 9918836 B/op	    5296 allocs/op
BenchmarkRegisteredPebbleWritePack/grants=10000-16       	       2	  25482308 ns/op	71556732 B/op	    5480 allocs/op
BenchmarkRegisteredPebbleWritePack/grants=100000-16      	       2	 146233306 ns/op	224073448 B/op	    7109 allocs/op
BenchmarkRegisteredPebbleWritePack/grants=1000000-16     	       2	1278815820 ns/op	1977869100 B/op	   21596 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=100-16         	       2	  13643225 ns/op	19086208 B/op	    8919 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=1000-16        	       2	  28076909 ns/op	 4548792 B/op	   78130 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=10000-16       	       2	 193263424 ns/op	63902932 B/op	  773594 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=100000-16      	       2	3138597479 ns/op	469357228 B/op	 7727621 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=1000000-16     	       2	53228893288 ns/op	4524379352 B/op	77267921 allocs/op

WritePack — baseline Pebble (commit 9676f15 = #874 tip + RFC doc)

BenchmarkRegisteredPebbleWritePack/grants=100-16         	       2	   7389587 ns/op	 4818044 B/op	    5868 allocs/op
BenchmarkRegisteredPebbleWritePack/grants=1000-16        	       2	  10892119 ns/op	 9232600 B/op	   14084 allocs/op
BenchmarkRegisteredPebbleWritePack/grants=10000-16       	       2	  48670153 ns/op	74173256 B/op	   95206 allocs/op
BenchmarkRegisteredPebbleWritePack/grants=100000-16      	       2	 390211097 ns/op	190087176 B/op	  908458 allocs/op
BenchmarkRegisteredPebbleWritePack/grants=1000000-16     	       2	4224277228 ns/op	2123730900 B/op	 9029143 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=100-16         	       2	  14696544 ns/op	19086420 B/op	    8917 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=1000-16        	       2	  27918710 ns/op	13946488 B/op	   78154 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=10000-16       	       2	 190792036 ns/op	54509180 B/op	  773565 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=100000-16      	       2	2967079472 ns/op	469359868 B/op	 7727622 allocs/op
BenchmarkRegisteredSQLiteWritePack/grants=1000000-16     	       2	53377326896 ns/op	4524378540 B/op	77267919 allocs/op

Read — both branches

# Autoresearch
BenchmarkRegisteredPebbleUnpackReadGrants/grants=100-16         	       2	   1837878 ns/op
BenchmarkRegisteredPebbleUnpackReadGrants/grants=1000-16        	       2	   3421652 ns/op
BenchmarkRegisteredPebbleUnpackReadGrants/grants=10000-16       	       2	  17417876 ns/op
BenchmarkRegisteredPebbleUnpackReadGrants/grants=100000-16      	       2	 128854950 ns/op
BenchmarkRegisteredSQLiteUnpackReadGrants/grants=100-16         	       2	   2156043 ns/op
BenchmarkRegisteredSQLiteUnpackReadGrants/grants=1000-16        	       2	   6251786 ns/op
BenchmarkRegisteredSQLiteUnpackReadGrants/grants=10000-16       	       2	  43168252 ns/op
BenchmarkRegisteredSQLiteUnpackReadGrants/grants=100000-16      	       2	 385242798 ns/op

# Baseline (commit 9676f153)
BenchmarkRegisteredPebbleUnpackReadGrants/grants=100-16         	       2	   2131626 ns/op
BenchmarkRegisteredPebbleUnpackReadGrants/grants=1000-16        	       2	   3770769 ns/op
BenchmarkRegisteredPebbleUnpackReadGrants/grants=10000-16       	       2	  16615650 ns/op
BenchmarkRegisteredPebbleUnpackReadGrants/grants=100000-16      	       2	 142946853 ns/op
BenchmarkRegisteredSQLiteUnpackReadGrants/grants=100-16         	       2	   2213450 ns/op
BenchmarkRegisteredSQLiteUnpackReadGrants/grants=1000-16        	       2	   6030950 ns/op
BenchmarkRegisteredSQLiteUnpackReadGrants/grants=10000-16       	       2	  42379082 ns/op
BenchmarkRegisteredSQLiteUnpackReadGrants/grants=100000-16      	       2	 386689590 ns/op