Skip to content

feat: graduate consumer offset replicas through durable range snapshots before it can serve#195

Merged
Migorithm merged 28 commits into
mainfrom
feat/offset-handoff
Jul 17, 2026
Merged

feat: graduate consumer offset replicas through durable range snapshots before it can serve#195
Migorithm merged 28 commits into
mainfrom
feat/offset-handoff

Conversation

@Migorithm

@Migorithm Migorithm commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Description:

Introduces Offset Placement Graduation (as outlined in the D9 design doc) to ensure that newly added replicas in a consumer offset range safely bootstrap historical checkpoint data before they become authoritative, without blocking live client traffic.

Previously, when a segment roll altered the active replica set, newly added replicas could temporarily serve empty or stale offset reads before receiving all historical data. This change strictly divorces replica placement from replica readiness.

Key Changes

  • Explicit Graduation Lifecycle (Joining -> Ready): Newly placed replicas are initialized in an unready Joining state. In this state, they cannot serve read requests or act as the leader. If queried as a leader, they safely reject the request with NotWriteLeader(None), forcing the client to gracefully retry.

  • Durable Range-Offset Bootstrapping: Existing Ready replicas transfer a complete snapshot of the current ledger slice (the latest positions for all consumer groups, including idle ones) to the joining replicas.

  • Concurrent Monotonic Merge: While the joining replica downloads the historical snapshot, it continues to receive live offset commits replicated by the active leader. These concurrent streams are safely reconciled using a monotonic merge (apply in ledger.rs), which ignores historical checkpoints if a newer live commit has already advanced the position.

  • Zero Client Penalty via Deferred Acknowledgement: Live client commits are routed to all replicas (including Joining ones), but the leader only requires durability acknowledgements from the Ready replicas to confirm success to the client, preserving cluster throughput and low latency.

  • Strict Durability Fence: A joining replica only graduates to Ready after both its bootstrap payload and a "bootstrap-completion marker" are safely durably flushed to the WAL, alongside any live commits received during the transition.

  • Fail-Closed Guarantees: If all Ready replicas are unavailable, the cluster will deliberately fail offset reads closed rather than returning incomplete data from a joining replica.

Testing

  • Added end-to-end simulation covering durable readiness recovery across restart (restarted_offset_replica_bootstraps_missed_epoch_before_commit_ack).

  • Added state machine tests ensuring unready leaders reject commits and publish requests (a_new_unready_leader_cannot_commit_or_publish_snapshots).

  • Added tests validating source eligibility, monotonic import concurrency, and completion marker durability.

@Migorithm Migorithm self-assigned this Jul 17, 2026
@Migorithm Migorithm added the documentation Improvements or additions to documentation label Jul 17, 2026
@Migorithm Migorithm added enhancement New feature or request feat labels Jul 17, 2026
@Migorithm
Migorithm merged commit 15be6dd into main Jul 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request feat

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant