Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#35

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-3020
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#35
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-3020

Conversation

@cursor

@cursor cursor Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

After v0.6.0 added HLC monotonic enforcement, restore_from_file() and gossip.receive() still assigned fresh HLC timestamps instead of preserving stored values.

Concrete scenario: Node A writes k=v1 with HLC (5000, 10, nodeA), snapshots, and restores after crash. A causal successor write k=v2 with HLC (5000, 11, nodeA) raises TimestampRegression and is silently dropped — post-disaster recovery loses writes. Gossip without HLC could also overwrite fresher data on existing keys.

Root cause

  • restore_from_file() rebuilt MemoryNode without the hlc field and never advanced the global HLC clock.
  • bulk_write() and gossip.receive() did not pass through HLC from wire payloads.

Fix

  • Add _parse_hlc() for snapshot/wire normalization (with legacy ts_ns fallback for pre-v0.6.0 snapshots).
  • Restore HLC on snapshot load and call _hlc.update() per node.
  • Pass HLC through bulk_write and gossip.receive; reject stale gossip updates and skip missing-HLC overwrites on existing keys.
  • Hold _lock during restore_from_file() and gc_expired() to prevent concurrent store corruption.

Validation

  • Reproduced pre-fix: restored HLC was fresh (wall, 0, uuid) and successor write failed.
  • Post-fix reproduction passes.
  • pytest tests/test_enterprise_backup.py tests/test_gossip.py tests/test_rust_brain.py tests/test_rust_brain_concurrency.py — 37 passed.
Open in Web View Automation 

After v0.6.0 added HLC monotonic enforcement, restore_from_file() and
gossip.receive() still assigned fresh HLC timestamps. Post-restore causal
writes were rejected (TimestampRegression) causing silent data loss; gossip
without HLC could overwrite fresher data.

- Add _parse_hlc() for snapshot/wire normalization with legacy fallback
- Restore HLC and advance global clock on snapshot restore
- Pass HLC through bulk_write and gossip receive with stale rejection
- Hold lock during restore_from_file and gc_expired to prevent races

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant