Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#23

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-adf3
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#23
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-adf3

Conversation

@cursor

@cursor cursor Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

Critical silent data loss in distributed deployments after snapshot restore.

v0.6.0 added Hybrid Logical Clock (HLC) monotonic enforcement on writes, but restore_from_file() and gossip.receive() did not preserve or apply HLC timestamps:

  1. After snapshot restore: restored nodes received fresh _hlc.now() timestamps instead of their saved HLC. Valid causal successor writes (e.g. gossip replay with hlc=(5000, 11, "nodeA")) were rejected as stale, causing silent data loss.
  2. Gossip without HLC: events missing hlc could overwrite fresher local data because remember() generated a new timestamp.
  3. Gossip with stale HLC: when hlc was present in the event but not passed through, stale remote values could overwrite fresher local data.

Root cause

restore_from_file() constructed MemoryNode without the hlc field from the snapshot. bulk_write() and gossip.receive() also omitted HLC passthrough, breaking causal ordering introduced in v0.6.0.

Fix

  • Add _parse_hlc() for snapshot/wire normalisation (with legacy ts_ns fallback for pre-v0.6.0 snapshots)
  • restore_from_file(): restore hlc per node and advance the global _hlc
  • bulk_write(): pass through hlc from row payloads
  • gossip.receive(): apply hlc, call update_hlc(), reject stale updates, skip missing-HLC overwrites on existing keys

Validation

  • New regression tests in test_enterprise_backup.py and test_gossip.py
  • Full suite: 192 passed, 8 skipped
Open in Web View Automation 

restore_from_file dropped hlc fields added in v0.6.0, assigning fresh
timestamps on restore. Post-crash causal writes and gossip replays with
pre-crash HLC were rejected (silent data loss), while gossip events
without HLC could overwrite fresher data.

- Add _parse_hlc() for snapshot/wire normalisation with legacy fallback
- restore_from_file: restore hlc and advance global _hlc per node
- bulk_write: pass through hlc from row payloads
- gossip.receive: apply hlc, reject stale updates, skip missing-hlc
  overwrites on existing keys
- Tests: snapshot HLC round-trip, gossip stale/new/missing-hlc cases

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant