Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#32

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-ec52
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#32
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-ec52

Conversation

@cursor

@cursor cursor Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

restore_from_file() and gossip.receive() did not preserve Hybrid Logical Clock (HLC) timestamps introduced in v0.6.0. After a snapshot restore:

  • Restored nodes received fresh HLC timestamps instead of their stored values
  • Causal successor writes with pre-crash HLC were rejected with TimestampRegressionsilent data loss
  • Gossip events without HLC could overwrite fresher data on existing keys

Concrete trigger: Snapshot a node with hlc=(5000, 10, "nodeA"), restore, then attempt remember("k", "v2", hlc=(5000, 11, "nodeA")) — the write fails because restore assigned a fresh wall-clock HLC.

Root cause

restore_from_file() reconstructed MemoryNode without reading the hlc field from snapshot JSON (defaults to _hlc.now()). gossip.receive() similarly omitted HLC passthrough and stale-update rejection.

Fix

  • Add _parse_hlc() for snapshot/wire normalisation with legacy ts_ns fallback
  • restore_from_file: restore hlc per node and advance global _hlc
  • bulk_write: pass through hlc from row payloads
  • gossip.receive: apply hlc, reject stale updates, skip missing-HLC overwrites on existing keys

Validation

  • Reproduced pre-fix: HLC not preserved, successor write raised TimestampRegression
  • Post-fix: HLC round-trip preserved, successor write succeeds
  • pytest tests/test_enterprise_backup.py tests/test_gossip.py tests/test_rust_brain.py — 28 passed
Open in Web View Automation 

restore_from_file dropped hlc fields added in v0.6.0, assigning fresh
timestamps on restore. Post-crash causal writes and gossip replays with
pre-crash HLC were rejected (silent data loss), while gossip events
without HLC could overwrite fresher data.

- Add _parse_hlc() for snapshot/wire normalisation with legacy fallback
- restore_from_file: restore hlc and advance global _hlc per node
- bulk_write: pass through hlc from row payloads
- gossip.receive: apply hlc, reject stale updates, skip missing-hlc
  overwrites on existing keys
- Tests: snapshot HLC round-trip, gossip stale/new/missing-hlc cases

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant