Skip to content

fix: gossip stale overwrite + async remember trust/tags#17

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-2854
Draft

fix: gossip stale overwrite + async remember trust/tags#17
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-2854

Conversation

@cursor

@cursor cursor Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

1. Gossip stale overwrite (data loss)

In distributed deployments using GossipProtocol, delayed gossip events could overwrite newer local memory writes.

Trigger: Node A writes key=state with a recent value. A stale gossip batch arrives from Node B (same key, older value) without ts_ns. receive() called remember() without a timestamp, which stamped wall-clock now() — newer than the local write — and silently replaced the correct value.

2. AsyncHiveStack.remember drops trust/tags (metadata loss / trust bypass)

AsyncHiveStack.remember() accepted trust and tags kwargs but only forwarded key and value to the underlying HiveStack. All async writes stored trust=1.0 and empty tags, breaking min_trust filtering and tag-based retrieval.

Root cause

  • Gossip: receive() did not propagate origin ts_ns, defeating RustBrain monotonic timestamp protection.
  • Async: run_in_executor(None, self._stack.remember, key, value) omitted keyword args.

Fix and validation

  • Gossip: skip updates for existing keys when ts_ns is missing; pass ts_ns when present so TimestampRegression rejects stale writes.
  • Async: delegate via lambda that forwards trust and tags.

Tests added:

  • test_gossip_without_ts_ns_does_not_overwrite_existing_key
  • test_gossip_with_older_ts_ns_is_rejected
  • test_gossip_with_newer_ts_ns_updates_existing_key
  • test_async_remember_preserves_trust_and_tags

Validation: 180 passed, 8 skipped

Open in Web View Automation 

Gossip receive() called remember() without ts_ns, so delayed gossip
events got wall-clock timestamps and could overwrite newer local writes.

- Skip gossip updates for existing keys when ts_ns is missing
- Pass ts_ns through when present so monotonic protection applies

AsyncHiveStack.remember() dropped trust and tags kwargs when delegating
to the sync stack via run_in_executor.

Regression tests added for both paths.

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant