feat(storage): add guarded JSONL tail repair#951
Closed
luoye520ww wants to merge 1 commit into
Closed
Conversation
Collaborator
|
Thank you for the guarded JSONL repair direction. I am closing this conflicting branch in favor of #975. The replacement was rebuilt on current Your original repair direction is credited in the replacement PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem\n\nA damaged final JSONL record can currently be detected, but there is no guarded way to repair that one deterministic failure without risking an overwrite of newer data.\n\n## Root cause\n\nA repair operation needs a stable snapshot, an explicit durable backup step, and an atomic replacement. A blind truncate or direct write can destroy valid history when another process writes between inspection and repair.\n\n## Scope\n\nThis PR adds a standalone JSONL tail repair service. It does not add a repair route, UI, automatic startup repair, or a backup storage implementation.\n\n## Changes\n\n- Inspect JSONL files with a bounded read and classify missing, valid, truncated, malformed, oversized, invalid UTF-8, and changed files.\n- Repair only a malformed final record when a valid prefix exists.\n- Require a caller-provided backup callback before any write.\n- Revalidate size, mtime, and SHA-256 before and after the backup callback.\n- Replace the file through a fsync'd temporary file and rename with retry; there is no direct-write fallback.\n- Verify the resulting bytes after replacement.\n\n## Behavior before\n\nA caller could detect a truncated tail but had no safe, reusable repair primitive.\n\n## Behavior after\n\nOnly the incomplete final record is removed. Interior corruption, races, missing backups, oversized files, and backup failures leave the original file untouched.\n\n## Safety\n\n- No raw content is returned in inspection results; the backup callback receives the bounded snapshot only because it is the explicit durability boundary.\n- The caller controls how sensitive snapshots are encrypted or stored.\n- The service refuses to repair without a backup callback.\n- All writes are bounded and atomic; verification failure is surfaced.\n\n## Typecheck\n\n
ext\nnpm.cmd run typecheck\nnpm.cmd --prefix kun run typecheck\n\n\nBoth passed.\n\n## Tests\n\next\nnpm.cmd --prefix kun test -- tests/thread-store-repair.test.ts\nnpm.cmd run lint\nnpm.cmd run build\ngit diff --check upstream/develop...HEAD\n\n\nResults: 5 targeted tests passed; root/Kun typecheck, lint, build, and diff checks passed.\n\n## Actual validation\n\nTests use real temporary files and exercise successful repair, backup failure, backup-time mutation, interior corruption, fully valid files, oversized files, and files without a valid prefix.\n\n## Non-goals\n\n- Automatic repair or startup mutation.\n- Backup encryption/storage.\n- SQLite index rebuild.\n- Renderer or IPC integration.\n\n## Issue\n\nPart of the thread-store integrity work.