StaticColdStore: add backfill mode for skipped-slot writes#79
Open
galadd wants to merge 1 commit into
Open
Conversation
Enables previously-skipped slots (offset == 0) to be filled in after the column's highest_written_slot has advanced past them. This unblocks ERA phase-2 reconstruction, which writes intermediate states at slots behind the era-boundary high-water mark. - New conf format (LHSTBLK3, 52 bytes) with allow_backfill flag, backfill_file_id, and backfill_data_len fields - Backward-compatible: LHSTBLK2 (36-byte) confs open read-only; upgrade to LHSTBLK3 happens automatically when allow_backfill is enabled - Column::put_backfill and Column::put_batch_backfill for single and batched backfill writes - heal_on_open handles crash recovery for both current and backfill files, including scan_and_zero_dangling_offsets - Enforced constraint: at most one non-current file per backfill batch - read_offset returns Ok(0) for missing offset files (required for backfill into sealed files) - StoreConfig::allow_backfill propagated to ColumnConfig - Full test suite: 19 tests covering open/get/put/put_batch, backfill behavior, crash recovery, conf upgrade, and edge cases
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Addressed
Resolves the monotonic-write blocker identified in #75 that prevents ERA phase-2 state reconstruction from working with the static cold backend.
After phase 1 of ERA import writes era-boundary states at slots 8192, 16384, etc., phase 2 (
reconstruct_states_parallel) backfills intermediate states at slots behindhighest_written_slot, which the static archive rejects:This PR enables those skipped slots to be filled in.
Proposed Changes
Backfill mode for StaticColdStore
When
allow_backfill = trueinStoreConfig, columns accept writes to previously-skipped slots (where the offset table entry is zero). Data is appended to the end of the data file (still append-only — no in-place writes), and the offset table entry for the backfilled slot is updated to point to it.highest_written_slotdoes not advance.Conf format upgrade (LHSTBLK2 → LHSTBLK3)
New 52-byte conf format adds:
flagsbyte (bit 0 =allow_backfill)backfill_file_idu64 — tracks which non-current file has uncommitted backfill databackfill_data_lenu64 — committed length of that fileOld 36-byte LHSTBLK2 confs open in read-only mode (backfill disabled). When
allow_backfill = trueis configured, the conf is automatically upgraded to LHSTBLK3 on first open. The magic bump prevents old code from opening a backfill-enabled store.Crash recovery
heal_on_openhandles three crash windows:current_data_lenis truncated; no offset points to itscan_and_zero_dangling_offsetscatches any offset pointing past the committed length and zeros itscan_and_zero_dangling_offsetsruns in O(SLOTS_PER_FILE) = O(8192) per file, only on open, only for at most two files (current + backfill).Constraint: one non-current file per batch
A single
put_batch_backfillcall may target at most one non-current file. This matches actual usage (ERA reconstruction writes one 8192-slot era at a time) and ensures the conf can track the backfill file for crash recovery.Bug fix:
read_offseton missing offset filesPreviously,
read_offsetcalledFile::opendirectly and would panic when the offset file didn't exist. Backfill into a sealed file whose offset file hasn't been created yet triggers this. Now returnsOk(0)(slot was skipped).Test suite: 19 tests
Additional Info
Reconstruction integration
The
reconstruct.rschange to allow reconstruction with backfill is not included in this PR — it depends on sigp#9273 (ERA files) being rebased onto this branch. The current PR only adds the storage-layer capability. The caller-side integration lands when the two branches merge.At-most-two-files invariant
The design tracks at most one non-current backfill file in the conf (
backfill_file_id/backfill_data_len). This means at most two files can have uncommitted data at any time: the current file and one backfill file. If a crash leaves orphaned data in a file that isn't tracked by either field, that data is harmless (no offset points to it) but wastes space until overwritten. A future improvement could scan all files on open to reclaim orphaned space, but this isn't necessary for correctness.ERA reconstruction workload
Phase 2 reconstruction writes states at slots 1–8191, then 8193–16383, etc. Each era's backfill targets exactly one non-current file before moving to the next. The one-non-current-file-per-batch constraint is naturally satisfied. If future workloads need to backfill across multiple sealed files in a single batch, the conf format would need a more general tracking mechanism (e.g., a separate backfill manifest file).