Storage Engine v4 — Stack 4: cross-engine compaction via IngestAndExcise#871
Open
c1-squire-dev[bot] wants to merge 1 commit into
Open
Conversation
c1-squire-dev
Bot
force-pushed
the
pquerna/storage-v4-stack4-compaction
branch
2 times, most recently
from
ebbb431 to
05ba86f
Compare
Implements the v3 cross-engine compaction primitive (RFC 0004 §3.10).
Given a `base` (destination) Pebble engine and a `source` (an applied
sync_run), produces a new `base` state where every key under syncID
matches source exactly — pre-existing destination data under that
syncID is excised and replaced atomically.
Mechanism per the micro-test in /tmp/baton-rfc-microtests/ingest_excise_test.go
(commit 8e91f3aa proved IngestAndExcise survives a 1000-row excise +
500-row ingest round-trip):
for each bucket (grant primary, by_entitlement, by_principal):
build SST from source's [sync_lo, sync_hi) range
base.IngestAndExcise(ctx, [sst_path], shared=nil, external=nil,
exciseSpan=[sync_lo, sync_hi))
Each call is atomic from base's perspective — concurrent readers see
old-OR-new for that range, never a mix.
New code:
pkg/synccompactor/pebble/compactor.go
- Compactor type, NewCompactor(base, tmpDir).
- Compact(ctx, source, syncID) — drives the per-bucket loop.
- Empty-source path: DeleteRange on destination (excise without
ingest) + return ErrEmptySync so callers can distinguish "source
was empty" from "compact succeeded with data".
- SST files cleaned up via defer even after Pebble's ingest links
them in (Pebble retains its own reference).
pkg/synccompactor/pebble/bucket_plans.go
- bucketPlan struct + buildBucketPlans(syncIDBytes) -> []bucketPlan.
- Stack 4 MVP covers grant + grant_by_entitlement +
grant_by_principal; other record-type buckets ship in the same
commit series as their Stack 3 record-type implementations.
pkg/synccompactor/pebble/sst_writer.go
- sstBuilder wraps pebble/sstable.Writer with the v3 table format
pinned (SDKPebbleFormat.MaxTableFormat). buildSSTFromIter drains
a pebble.Iterator into a sorted SST.
Exports added to engine:
pkg/dotc1z/engine/pebble/engine.go
- (*Engine).DB() *pebble.DB — accessor for the compactor.
pkg/dotc1z/engine/pebble/keys.go
- GrantSyncLowerBound / UpperBound
- GrantByEntitlementSyncLowerBound / UpperBound
- GrantByPrincipalSyncLowerBound / UpperBound
These are exported for the synccompactor package; not part of the
stable Engine public API (per godoc).
Tests (5/5 passing):
- TestCompactBasicRoundtrip — 200 grants moved src → dst.
- TestCompactReplacesExisting — dst has 500 stale grants; src has
50 fresh; post-compact dst has exactly 50 (and 0 stale-ent index
entries — proves index excision works).
- TestCompactIsolatesOtherSyncs — dst has sync_A + sync_B (100+100);
src has 25 under sync_A; post-compact: dst sync_A = 25, dst
sync_B = 100 (untouched). This is the critical safety property.
- TestCompactEmptySource — empty src wipes dst's range and returns
ErrEmptySync.
- TestCompactBadInputs — nil source, empty syncID, nil base all
return errors instead of panicking.
Refs: RFC v4 §3.10, Appendix C
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
c1-squire-dev
Bot
force-pushed
the
pquerna/storage-v4-stack4-compaction
branch
from
05ba86f to
66a8025
Compare
c1-squire-dev
Bot
force-pushed
the
pquerna/storage-v4-stack3-pebble-engine
branch
from
c898e42 to
4b2ee1a
Compare
Contributor
Author
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stack 4 of RFC 0004. Stacks on Stack 3.
The v3 cross-engine compaction primitive (RFC §3.10). Given a
base(destination) Pebble engine and asource(an applied sync), produces a newbasestate where every key under syncID matches source exactly — pre-existing destination data under that syncID is excised and replaced atomically.Mechanism (validated by the IngestAndExcise micro-test that was preserved at
/tmp/baton-rfc-microtests/ingest_excise_test.go):Each call is atomic from base's perspective — concurrent readers see old-OR-new for the range, never a mix. Zero proto encode/decode per record; zero LSM compaction churn from a Put loop.
New code at
pkg/synccompactor/pebble/:compactor.go—Compactor+NewCompactor(base, tmpDir)+Compact(ctx, source, syncID). Per-bucket loop with the empty-source path falling back toDeleteRange+ returningErrEmptySync.bucket_plans.go—buildBucketPlans(syncIDBytes) → []bucketPlan. Stack 4 MVP covers GrantRecord buckets; other record-type buckets ship alongside their Stack 3 implementations.sst_writer.go—sstBuilderwrappingpebble/sstable.Writerwith the v3 table format pinned (SDKPebbleFormat.MaxTableFormat).buildSSTFromIterdrains apebble.Iteratorinto a sorted SST.compactor_test.go— 5 cases.The critical test is
TestCompactIsolatesOtherSyncs: dst holds 100 grants under sync_A and 100 under sync_B, source has 25 under sync_A; post-compact dst has 25 under sync_A (matching source) and 100 under sync_B (untouched).Test plan
make lintclean🤖 Generated with Claude Code