fix(merge): recognize batch-existing.json so incremental updates don't drop ~75% of nodes#403
Open
atlas-architect wants to merge 1 commit into
Conversation
…t drop ~75% of nodes During an incremental `/understand` update the skill writes the pruned existing-graph payload as `batch-existing.json` alongside the freshly analyzed `batch-<N>.json` files. The merge step bucketed it by the `batch-(\d+)` filename match, so `batch-existing.json` (no digits) fell through to "unrecognized" and was silently dropped at load — only a stderr warning, exit 0. Net effect: incremental merges lost every carried-over node. In a 305-node baseline + 10 changed files, the incremental output had 92 nodes; the 213 unchanged-file nodes were gone. Fix: recognize `existing` as a valid logical batch (sorted/bucketed as index -1 so it loads before the numbered batches). Adds a regression test (TestIncrementalExistingBatch) that runs the script end-to-end and asserts batch-existing.json nodes survive the merge. Fixes Egonex-AI#292 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
/understand's incremental-update path is silently lossy. During an incremental update the skill writes the pruned existing-graph payload asbatch-existing.jsonalongside the freshly-analyzedbatch-<N>.jsonfiles, then runsmerge-batch-graphs.py.But the merge step buckets files by the
batch-(\d+)filename match, sobatch-existing.json(no digits) falls through to the "unrecognized" branch and is dropped at load — only a stderr warning, exit 0. Net effect: every carried-over node from the previous scan is lost.In a 305-node baseline + 10 changed files, the incremental output had 92 nodes — the 213 unchanged-file nodes were gone. (The known workaround was to manually rename
batch-existing.json→batch-0.json.)This is the bug tracked in #292.
Fix
Recognize
existingas a valid logical batch inmerge-batch-graphs.py:batch-(\d+|existing)(?:-part-(\d+))?\.jsonbatch-existing.jsonis bucketed/sorted as logical index-1so it loads before the freshly-analyzed numbered batchesSo
batch-existing.jsonis now loaded and merged like any other batch instead of being dropped. Genuinely-malformed filenames (e.g.batch-fused-8-13.json) still hit the existing "unrecognized → warn" path, unchanged.Test
Adds
TestIncrementalExistingBatchtotests/skill/understand/test_merge_batch_graphs.py— runs the script end-to-end on a temp intermediate dir containingbatch-existing.json+batch-0.jsonand asserts the existing nodes survive the merge (and aren't reported as a dropped/unrecognized filename). Full suite green locally (71 tests).Surfaced during fleet-side dogfooding (Atlas Intelligence — same project that filed #292 and the emoji-folder
git ls-files -zfix in #231).Fixes #292