feat: graduate consumer offset replicas through durable range snapshots before it can serve#195
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Introduces Offset Placement Graduation (as outlined in the D9 design doc) to ensure that newly added replicas in a consumer offset range safely bootstrap historical checkpoint data before they become authoritative, without blocking live client traffic.
Previously, when a segment roll altered the active replica set, newly added replicas could temporarily serve empty or stale offset reads before receiving all historical data. This change strictly divorces replica placement from replica readiness.
Key Changes
Explicit Graduation Lifecycle (Joining -> Ready): Newly placed replicas are initialized in an unready Joining state. In this state, they cannot serve read requests or act as the leader. If queried as a leader, they safely reject the request with NotWriteLeader(None), forcing the client to gracefully retry.
Durable Range-Offset Bootstrapping: Existing Ready replicas transfer a complete snapshot of the current ledger slice (the latest positions for all consumer groups, including idle ones) to the joining replicas.
Concurrent Monotonic Merge: While the joining replica downloads the historical snapshot, it continues to receive live offset commits replicated by the active leader. These concurrent streams are safely reconciled using a monotonic merge (apply in ledger.rs), which ignores historical checkpoints if a newer live commit has already advanced the position.
Zero Client Penalty via Deferred Acknowledgement: Live client commits are routed to all replicas (including Joining ones), but the leader only requires durability acknowledgements from the Ready replicas to confirm success to the client, preserving cluster throughput and low latency.
Strict Durability Fence: A joining replica only graduates to Ready after both its bootstrap payload and a "bootstrap-completion marker" are safely durably flushed to the WAL, alongside any live commits received during the transition.
Fail-Closed Guarantees: If all Ready replicas are unavailable, the cluster will deliberately fail offset reads closed rather than returning incomplete data from a joining replica.
Testing
Added end-to-end simulation covering durable readiness recovery across restart (restarted_offset_replica_bootstraps_missed_epoch_before_commit_ack).
Added state machine tests ensuring unready leaders reject commits and publish requests (a_new_unready_leader_cannot_commit_or_publish_snapshots).
Added tests validating source eligibility, monotonic import concurrency, and completion marker durability.