Skip to content

feat(storage): add guarded JSONL tail repair#951

Closed
luoye520ww wants to merge 1 commit into
KunAgent:developfrom
luoye520ww:codex/thread-store-repair-standalone
Closed

feat(storage): add guarded JSONL tail repair#951
luoye520ww wants to merge 1 commit into
KunAgent:developfrom
luoye520ww:codex/thread-store-repair-standalone

Conversation

@luoye520ww

Copy link
Copy Markdown
Collaborator

Problem\n\nA damaged final JSONL record can currently be detected, but there is no guarded way to repair that one deterministic failure without risking an overwrite of newer data.\n\n## Root cause\n\nA repair operation needs a stable snapshot, an explicit durable backup step, and an atomic replacement. A blind truncate or direct write can destroy valid history when another process writes between inspection and repair.\n\n## Scope\n\nThis PR adds a standalone JSONL tail repair service. It does not add a repair route, UI, automatic startup repair, or a backup storage implementation.\n\n## Changes\n\n- Inspect JSONL files with a bounded read and classify missing, valid, truncated, malformed, oversized, invalid UTF-8, and changed files.\n- Repair only a malformed final record when a valid prefix exists.\n- Require a caller-provided backup callback before any write.\n- Revalidate size, mtime, and SHA-256 before and after the backup callback.\n- Replace the file through a fsync'd temporary file and rename with retry; there is no direct-write fallback.\n- Verify the resulting bytes after replacement.\n\n## Behavior before\n\nA caller could detect a truncated tail but had no safe, reusable repair primitive.\n\n## Behavior after\n\nOnly the incomplete final record is removed. Interior corruption, races, missing backups, oversized files, and backup failures leave the original file untouched.\n\n## Safety\n\n- No raw content is returned in inspection results; the backup callback receives the bounded snapshot only because it is the explicit durability boundary.\n- The caller controls how sensitive snapshots are encrypted or stored.\n- The service refuses to repair without a backup callback.\n- All writes are bounded and atomic; verification failure is surfaced.\n\n## Typecheck\n\n ext\nnpm.cmd run typecheck\nnpm.cmd --prefix kun run typecheck\n\n\nBoth passed.\n\n## Tests\n\n ext\nnpm.cmd --prefix kun test -- tests/thread-store-repair.test.ts\nnpm.cmd run lint\nnpm.cmd run build\ngit diff --check upstream/develop...HEAD\n\n\nResults: 5 targeted tests passed; root/Kun typecheck, lint, build, and diff checks passed.\n\n## Actual validation\n\nTests use real temporary files and exercise successful repair, backup failure, backup-time mutation, interior corruption, fully valid files, oversized files, and files without a valid prefix.\n\n## Non-goals\n\n- Automatic repair or startup mutation.\n- Backup encryption/storage.\n- SQLite index rebuild.\n- Renderer or IPC integration.\n\n## Issue\n\nPart of the thread-store integrity work.

@XingYu-Zhong

Copy link
Copy Markdown
Collaborator

Thank you for the guarded JSONL repair direction. I am closing this conflicting branch in favor of #975.

The replacement was rebuilt on current develop with incremental bounded scanning and an explicit concurrency contract. It requires a caller-supplied maintenance runner around the entire inspect/backup/replace/verify transaction, preserves an exact backup, uses a synced exclusive temporary file and original-inode guard, and returns changed/a retained conflict path whenever a safe version cannot be proven. The API fails immediately when writer exclusion is not supplied; all writers for that path must cooperate with the same runner.

Your original repair direction is credited in the replacement PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants