[Bug] build_merge severs edges from UNCHANGED files into re-extracted files (−12% doc↔code in one update cycle): endpoint nodes are replaced, the surviving edges dangle, and the dangling-edge drop silently eats them

[gkaganas-mparticle]

Version measured: graphifyy 0.8.46, one real incremental update on a
112,714-node / 258,776-edge monorepo graph. Code references below are
against current v8 (0.9.8) — the mechanism is unchanged there. Note: the
hyperedge half of what we measured was since fixed by #1574 (4f40967);
this report is about the ordinary-edge half, plus one residual hyperedge gap.

What we observed (one --update cycle, 298 changed files)

Metric	Before	After	Delta
doc↔code cross-edges	3,055	2,682	−12.2%
… lost			526 (of which 497 had an endpoint node replaced by the merge)
… gained			153
hyperedges (on 0.8.46, pre-#1574)	240	14	−94%; 182/240 had fully resolvable member sets in the merged graph

The worst losers were docs that did not change: e.g. one architecture doc
went from 189 cross-edges to 68 purely because the code on the other end was
re-extracted. Repeat updates compound this — each cycle erodes another slice
of exactly the high-value cross-file linkage.

Mechanism

build_merge (v8 build.py ~735) implements replace-per-source: every
source_file present in the new chunks is dropped from the loaded base —
nodes and edges — before merging (~lines 816–820):

existing_nodes = [n for n in existing_nodes if _kept(n)]
existing_edges = [e for e in existing_edges if _kept(e)]

That filter is keyed on the edge's own source_file. An edge extracted from
an unchanged doc (source_file = doc/foo.md) pointing at a node from
src/bar.py survives the filter — correctly, its provenance didn't
change. But src/bar.py was re-extracted, so its old nodes were dropped and
its new nodes may carry different IDs (LLM extraction is not ID-stable across
runs — see #1695 — and granularity/labels shift). The surviving doc edge now
references a node ID that no longer exists.

Those dangling edges then die silently in build_from_json (v8 build.py
~562–567):

if src not in node_set:
    src = norm_to_id.get(_normalize_id(src), src)
...
if src not in node_set or tgt not in node_set:
    continue  # skip edges to external/stdlib nodes - expected, not an error

The norm_to_id remap only rescues casing/punctuation variants. A genuinely
new ID for the same symbol (different qualifier, different chunk naming) is
not rescued, and the drop is deliberately unlogged because the same code path
legitimately absorbs stdlib/external references. So the erosion is invisible:
no warning, node counts grow, and only a cross-edge count comparison exposes
it. (This is the "local destruction while the graph grows globally" failure
mode #1652 describes from the node side; here it's the edge side.)

Residual hyperedge gap (post-#1574)

#1574's fix carries forward hyperedges keyed on source_file — unchanged
files' hyperedges survive, re-extracted files' are dropped on the expectation
the new chunk re-emits them. Two gaps remain:

1. A re-extracted file's new chunk frequently does not re-emit an
   equivalent hyperedge (same non-determinism as above) — the old one is
   dropped, nothing replaces it.
2. A carried hyperedge's nodes list can reference replaced node IDs, so
   it survives as an orphan (or is dropped downstream), even when every
   member still exists under a new ID.

In our measured cycle, 182 of the 240 original hyperedges had member sets
that were still fully resolvable in the merged graph — the information
needed to keep them was present; only the identity linkage was lost.

Proposed fix

Preserve what provably survives, keyed on resolvability rather than
provenance:

1. Edge preservation pass in build_merge, after the merged node set is
   final: for each pre-existing edge that was kept by the source_file
   filter but now dangles, attempt to re-resolve each dead endpoint against
   the merged graph — exact ID, then _normalize_id, then the existing
   ghost-merge key (source_file basename, label) that build_from_json
   already uses for AST/LLM twin detection (v8 ~lines 435–470). Rewrite the
   endpoint and keep the edge if exactly one candidate matches; drop (and
   count) otherwise.
2. Hyperedge preservation: for dropped/carried hyperedges, re-resolve
   member lists the same way; carry any hyperedge whose full member set
   survives (by resolved ID), id-deduped via the existing
   attach_hyperedges.
3. Make the loss visible: build_merge already prints prune counts;
   print severed-edge and dropped-hyperedge counts too, so a 497-edge
   silent loss becomes one loud line. (Pairs with Feature: build_merge dry-run, god-node degree-drop alert, and pre-write backup #1652's dry-run/alert
   proposal.)

Conservative matching (unique-candidate-only) keeps this from inventing
edges; everything it rescues is an edge/hyperedge the graph had, whose both
endpoints still exist.

Minimal repro sketch

from graphify.build import build, build_merge
from graphify.export import to_json

doc_chunk = {"nodes": [
    {"id": "payment_flow", "label": "Payment Flow", "file_type": "document",
     "source_file": "doc/payments.md"},
    {"id": "charge_card", "label": "charge_card", "file_type": "code",
     "source_file": "src/pay.py"}],
  "edges": [{"source": "payment_flow", "target": "charge_card",
             "relation": "references", "source_file": "doc/payments.md"}]}

G = build([doc_chunk]); to_json(G, "graphify-out/graph.json")
# assert: 1 doc→code edge

# Re-extract ONLY src/pay.py; same function, new node ID (qualified this time)
new_chunk = {"nodes": [
    {"id": "pay_charge_card", "label": "charge_card", "file_type": "code",
     "source_file": "src/pay.py"}], "edges": []}

G2 = build_merge([new_chunk])
# doc/payments.md was NOT re-extracted, charge_card still exists (as
# pay_charge_card, same label, same file) — but the doc→code edge is gone.

Same shape with a hyperedge whose nodes include the renamed ID reproduces
the residual hyperedge gap post-#1574.

Related

• build_merge / --update drops existing graph's hyperedges from unchanged files (only nodes+edges are read from graph.json) #1574 (fixed) — hyperedge carry-forward by source_file; this report is
  the by-resolvability completion of it.
• Feature: build_merge dry-run, god-node degree-drop alert, and pre-write backup #1652 — dry-run / degree-drop alert; item 3 above is the minimal in-tree
  version of that visibility.
• Non-deterministic graphs from identical inputs: extraction lacks seed/prompt-override controls and extract_corpus_parallel ignores extraction-spec #1695 — extraction non-determinism is the amplifier here: the less
  ID-stable extraction is, the more the merge severs.
• --update: build_merge(prune_sources=...) silently no-ops when root is not passed (absolute vs relative source_file mismatch, Windows) #1151 — a different way build_merge's source_file keying silently
  misbehaves.
• [RFC] Native graphify update for the semantic layer — a scripted conductor with a pluggable LLM hook (we measured: the math is 0.89s–52s, the agent-walked workaround is 11–14 min) #1710 — our native-update RFC folds this preservation pass into
  the proposed conductor's merge step; [Feature Request] Native sharded-graph loading: when graph.json is absent but graph.json.part-* + a shard manifest exist, reconstitute transparently (query/path/explain/affected/serve/MCP) #1708 is the same
  committed-graph workflow this erosion degrades.

Offer

We have the forensic tooling (before/after cross-edge diff by source doc,
endpoint-replacement classification) and are implementing the preservation
pass downstream. Happy to contribute the PR with regression tests pinned to
the mechanism (unchanged-doc edge into renamed code node survives; ambiguous
candidate is dropped; counts are printed) if the approach works for you.