feat(rollout): MemWAL ingest with server-id sharding#126
Merged
Conversation
Convert RolloutStore from atomic Dataset::append to Lance MemWAL, sharded by stable server instance id. Each REST server instance owns exactly one MemWAL shard (shard_uuid = uuid_v5(NAMESPACE_OID, instance_id)), so the single-active-writer + epoch-fencing invariant holds by construction and a plain round-robin load balancer is correct — no consistent hashing, lock service, or client changes. close-per-append flushes each write to object storage before returning, and the LSM read path rebuilds from every shard's manifests, so any instance sees every instance's writes (read-your-writes with no affinity). Store binary_payload inline instead of blob-v2 offloaded. The MemWAL LSM read path is the only read path for rollouts and has no blob-materialization step, so an offloaded (lance-encoding:blob) column reads back as None and get_blob / the blob endpoint would return nothing. Inline LargeBinary round-trips correctly; list/get_by_id still project the column out so hot-path scans never materialize artifact bytes (learner doesn't pay for artifacts), and get_blob projects it in on demand. Removes the broken blob_columns knob from the rollout options, server handler, unified facade, and CreateRolloutStoreRequest. Thread instance id through the server: --instance-id / INSTANCE_ID, falling back to HOSTNAME (StatefulSet pod ordinal), into AppState and the rollout shard key. Reproducibility is now a filter over immutable append-only rows (e.g. policy_version), not a per-add checkout snapshot — MemWAL appends do not advance the base dataset version. Add specs/rollout-deployment.md covering the K8s StatefulSet deployment, the sharding model, and the single-writer/read-your-writes/append-only rationale. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Apply cargo fmt to the MemWAL writer/index-init call chains added in the previous commit; satisfies the `cargo fmt --all -- --check` CI gate. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
4 tasks
beinan
added a commit
that referenced
this pull request
Jul 10, 2026
…table (#127) ## Summary Follow-up to #126. In 0.6.0, rollout MemWAL appends accumulate flushed generations under `_mem_wal/{shard}/` **forever** — there was no path to fold them back into the base table, so read cost grew linearly with append count (every `list`/`get` unions all generations). This PR adds an opt-in, **size-triggered self-merge** (the "external compactor" path Lance's MemWAL LSM design anticipates). Set `--rollout-merge-after-generations N` (env `ROLLOUT_MERGE_AFTER_GENERATIONS`, or `RolloutStoreOptions::merge_after_generations`). After an append leaves this instance's **own** shard with ≥ N un-merged generations, the same `add` synchronously: 1. reads each flushed generation (a self-contained Lance dataset under `_mem_wal/{shard}/`), 2. `Dataset::append`s their rows into the **base table**, and 3. `commit_update`s the shard manifest to drain `flushed_generations` to empty — keeping `replay_after_wal_entry_position` so a reopened writer never re-replays merged WAL entries. `N = 0` (default) preserves the 0.6.0 pure-accumulation behavior. ## Why it's safe - **No peer fencing.** The merge `claim_epoch`s the shard, bumping its writer epoch — but each instance merges *only the shard it writes*, which has no other live writer under the server-id sharding model (#126 / spec §3). The instance's next `add` re-claims the current epoch. - **Crash-safe.** Rollout rows are immutable and deduped by `id` at read time. A crash between step 2 and step 3 (rows in base, manifest not yet drained) just means a reader sees the rows via *both* base and the still-listed generation and dedups them — no double counting. The next merge drains the manifest. - **No coordinator.** Self-merge per instance: no cronjob, no admin endpoint, no cross-instance lock. The load balancer keeps spraying appends; each instance keeps its own shard compact. ## Benchmark `bench_merge_read_amplification` (200 single-row generations, local FS, release): | Read (`list` scan of 200 rows) | Latency | |---|---| | over 200 un-merged generations | ~654 ms | | after merge into base table | ~50 ms | | **speedup** | **~13×** | Slowest single `add` (merge-on-every-append at N=1): ~20 ms. ## Changes - `RolloutStoreOptions::merge_after_generations` + `RolloutStore::{maybe_merge_own_shard, merge_own_shard}`, triggered in `add()` after `close()`. - Server: `--rollout-merge-after-generations` flag → `AppState` → rollout route. - Threaded through the unified facade (`unified_rollout.rs`). - `specs/rollout-deployment.md` §6.1 documents the mechanism, safety, trade-off, and benchmark; §9 checklist updated. ## Test plan - [x] `below_threshold_no_merge` — generations accumulate, no merge under threshold - [x] `merge_at_threshold_drains_shard_and_preserves_reads` — shard drains, rows present exactly once, inline artifact bytes still fetchable post-merge - [x] `merge_is_visible_to_a_fresh_reader_instance` — merged rows land in base, visible to a newly-opened reader - [x] core suite (106 pass) + server suite (26 pass), clippy clean, fmt clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Beinan Wang <beinanwang@microsoft.com> Co-authored-by: Claude Opus 4 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Converts
RolloutStorefrom atomicDataset::appendto Lance MemWAL, sharded by stable server instance id, so rollout ingest can scale across multiple REST server instances behind a plain load balancer.shard_uuid = uuid_v5(NAMESPACE_OID, instance_id)). No two instances share a shard, so MemWAL's single-active-writer + epoch-fencing invariant holds by construction — a round-robin LB is correct, with no consistent hashing, lock service, or client changes.close-per-append flushes each write to object storage beforeaddreturns; the LSM read path rebuilds from every shard's manifests, so any instance sees every instance's writes.--instance-id/INSTANCE_ID, falling back toHOSTNAME(StatefulSet pod ordinal), flows intoAppStateand the rollout shard key.Blob fix (was a latent production bug)
RolloutStoredefaultedbinary_payloadto blob-v2 offload, but the MemWAL LSM read path — the only read path for rollouts — has no blob-materialization step. Confirmed three ways:ContextStore's identicalget_blobonly round-trips because its default is inline; zeroblobreferences anywhere in Lance'smem_walmodule;LsmScanner's public API has no blob method. So offloaded artifacts read back asNoneandfetch_rollout_blobwould 404 in production.Fix: store
binary_payloadinline (LargeBinary). The "learner doesn't pay for artifacts" property is preserved columnar-ly —list/get_by_idproject the column out;get_blobprojects it in on demand. Removes the brokenblob_columnsknob from the rollout options, server handler, unified facade, andCreateRolloutStoreRequest.Reproducibility
MemWAL appends do not advance the base dataset version, and rollout rows are append-only/immutable, so per-checkpoint reproducibility is a filter over immutable rows (e.g.
policy_version), not a per-addcheckoutsnapshot.Docs
Adds
specs/rollout-deployment.md: K8s StatefulSet deployment, the sharding model, and the single-writer / read-your-writes / append-only rationale.Test plan
cargo test -p lance-context-core rollout— 5/5 pass, incl.distinct_shards_share_one_dataset(two instances write distinct shards of one dataset, both visible to any reader)cargo test -p lance-context-server— 26/26 pass, incl. end-to-endfetch_rollout_blobHTTP paths (JSON + multipart) returning real artifact bytes through the MemWAL scannercargo test -p lance-context-core(103) /-p lance-context(6) — greencargo build --workspaceincl. Python bindings — cleancargo clippyon touched crates — cleanNote for reviewers
Removed
blob_columnsfromCreateRolloutStoreRequest(the rollout API). Old clients still sending it are unaffected (serde drops unknown fields); the only working value was inline anyway. The separate context store'sblob_columnsis untouched.🤖 Generated with Claude Code