fix: gossip stale overwrite + async remember trust/tags#17
Draft
cursor[bot] wants to merge 1 commit into
Draft
Conversation
Gossip receive() called remember() without ts_ns, so delayed gossip events got wall-clock timestamps and could overwrite newer local writes. - Skip gossip updates for existing keys when ts_ns is missing - Pass ts_ns through when present so monotonic protection applies AsyncHiveStack.remember() dropped trust and tags kwargs when delegating to the sync stack via run_in_executor. Regression tests added for both paths. Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug and impact
1. Gossip stale overwrite (data loss)
In distributed deployments using
GossipProtocol, delayed gossip events could overwrite newer local memory writes.Trigger: Node A writes
key=statewith a recent value. A stale gossip batch arrives from Node B (same key, older value) withoutts_ns.receive()calledremember()without a timestamp, which stamped wall-clocknow()— newer than the local write — and silently replaced the correct value.2. AsyncHiveStack.remember drops trust/tags (metadata loss / trust bypass)
AsyncHiveStack.remember()acceptedtrustandtagskwargs but only forwardedkeyandvalueto the underlyingHiveStack. All async writes storedtrust=1.0and empty tags, breakingmin_trustfiltering and tag-based retrieval.Root cause
receive()did not propagate origints_ns, defeating RustBrain monotonic timestamp protection.run_in_executor(None, self._stack.remember, key, value)omitted keyword args.Fix and validation
ts_nsis missing; passts_nswhen present soTimestampRegressionrejects stale writes.trustandtags.Tests added:
test_gossip_without_ts_ns_does_not_overwrite_existing_keytest_gossip_with_older_ts_ns_is_rejectedtest_gossip_with_newer_ts_ns_updates_existing_keytest_async_remember_preserves_trust_and_tagsValidation:
180 passed, 8 skipped