Autoresearch/pebble perf 20260525#876

Closed

pquerna wants to merge 34 commits into

mainfrom

autoresearch/pebble-perf-20260525

Commits on May 24, 2026

feat(storage v3): combined v4 storage-engine PR stack (for macro review)

pquerna
and
claude
committed

Commits on May 25, 2026

fix(storage): address v4 review comments
pquerna
committed
perf(storage v3): fat-batch Put*Records + fresh-sync NoSync write path

pquerna
and
claude
committed
feat(storage v3): wire real pagination through adapter List* methods

pquerna
and
claude
committed
feat(storage v3): close syncer-blocking gaps (A) + IfNewer (D) + Stats (E)

pquerna
and
claude
committed
fix(storage v3): close reviewer-bot findings (G) — error propagation, cache leak, Fatalf

pquerna
and
claude
committed
fix(storage v3): always read-before-write on PutXRecords (no fresh-sync skip)

pquerna
and
claude
committed
fix(ci): pin golangci-lint to v2.12.2 + restore nolint directives

pquerna
and
claude
committed
fix(go.mod): pin Go version to 1.25.2 to match CI

pquerna
and
claude
committed
fix(dotc1z): tiebreak getFinishedSync ORDER BY for Windows time resolution

pquerna
and
claude
committed
docs(rfc 0004): autoresearch plan for Pebble engine perf loop

pquerna
and
claude
committed
autoresearch: scaffold Pebble engine perf loop
pquerna
committed
fix(equivalence/test): silence gosec G602 false positive
pquerna
committed
P1.2b: L0CompactionThreshold 2→8. writepack_1m 4245.86→3575.28ms (-15.8%); writepack_100k -6.3%; reads & SQLite sentinel within tolerance. Solo write +21% in ns/op is engine startup noise on a non-…
pquerna
committed
P3.8: scratch byte buffers reused across PutGrantRecords loop + proto.MarshalAppend instead of Marshal. writepack_1m 3575→3375ms (-5.6%). Allocs/op 9.0M→4.0M (-55%). Bytes/op 2108→1654 MB (-21.5%).…
pquerna
committed
Hoist resolveSyncBytes out of PutGrantRecords loop with last-value cache. writepack_1m 3375→3209ms (-4.9%). Allocs 4.02M→3.02M (-25%) — the 1M per-record 20-byte syncBytes copies are gone. Falls ba…
pquerna
committed
Split PutGrantRecords into two pebble.Batches: primary keys vs index keys. writepack_1m 3209→2798ms (-12.8%). Primary keys arrive sorted by external_id (the iteration order); pdqsort detects this a…
pquerna
committed
Use NewBatchWithSize(len*600) and len*140 to pre-allocate the priBatch and idxBatch internal buffers. writepack_1m 2798→2626ms (-6.1%). bytes_op 1578→1208 MB (-23%). Pebble batches grow by 2x when …
pquerna
committed
Skip read-before-write Get during the first PutGrantRecords call of a fresh sync. writepack_1m 2626→2244ms (-14.5%). Mechanism: within a single PutGrantRecords call, db.Get queries the DB but batch…
pquerna
committed
Parallel-build PutGrantRecords for fresh-sync skipGet path with len(records)>=256. Two goroutines build priBatch and idxBatch concurrently. writepack_1m 2244→2046ms (-8.8%); writepack_100k 222→205 …
pquerna
committed
autoresearch.md: document loop results (-52.3% from RFC baseline)
pquerna
committed
4-way shard the priBatch build in the parallel skipGet path. Each shard goroutine builds a local pebble.Batch (NewBatchWithSize hint); main goroutine Apply's them in order into the final priBatch. …
pquerna
committed
autoresearch.md: update with priBatch sharding win (-56.1% cumulative)
pquerna
committed
autoresearch.md: capture noise-floor calibration + parallel-close dead end
pquerna
committed
Pre-sort the idxBatch via 4-way parallel sort + k-way merge before Setting entries. writepack_1m 1884→1656ms (-12.1%); 100k -9.1%; 10k -21.4%. The idxBatch's flushable-batch promotion sort over 2M …
pquerna
committed
Arena-style storage for the parallel idx-key sort. Replaces the 2M individual `make([]byte, 0, 96)` allocations from the prior commit with a per-shard (data []byte, bounds [][2]uint32) pair \u2014 …
pquerna
committed
Move priBatch.Commit and idxBatch.Commit INSIDE their respective parallel goroutines (fresh-sync skipGet path). writepack_1m 1539\u21921455 ms (-5.4%). 10k workload -13.1%. Mechanism is richer than…
pquerna
committed
After both parallel commits, call e.db.AsyncFlush() to force Pebble's flusher to schedule immediately rather than wait for its next polling cycle. writepack_1m 1455\u21921420 ms (-2.4%, borderline …
pquerna
committed
Replace V2GrantToV3's per-grant builder allocations with a per-call grantTranslateArena (3 pre-sized slices for GrantRecord/EntitlementRef/PrincipalRef). writepack_1m 1420\u21921352 ms (-4.8%). wri…
pquerna
committed
Parallelize the V2\u2192V3 grant translation in Adapter.PutGrants with 4 shard workers, each writing to a disjoint range of the records slice with its own arena. Previously this 1M-iteration loop r…
pquerna
committed
Async tmpdir cleanup in registeredStore.Close: spawn os.RemoveAll in a goroutine instead of blocking. The deferred RemoveAll was synchronously removing the per-store temp directory (Pebble engine d…
pquerna
committed
autoresearch.ideas.md: capture closed axes (do-not-retry list)
pquerna
committed
autoresearch.ideas.md: consolidate WriteEnvelope parallel-read attempts as closed
pquerna
committed
autoresearch.ideas.md: close parallel-large-alloc axis (3 attempts confirmed dead)
pquerna
committed