[Feature Request] Native sharded-graph loading: when graph.json is absent but graph.json.part-* + a shard manifest exist, reconstitute transparently (query/path/explain/affected/serve/MCP)

[gkaganas-mparticle]

Version: graphifyy 0.8.46 (behavior re-verified against v8 @ 0.9.8 —
load paths unchanged in the relevant way). Env: committed-graph workflow;
GitHub is the system of record for graphify-out/.

Problem

At monorepo scale, graph.json outgrows GitHub's hard 100 MB per-file blob
limit. Ours is ~125 MB (112,714 nodes / 258,776 links), so it cannot be
committed raw. But committing the graph is the whole point of a
CI-maintained knowledge graph: every clone, worktree, and CI job should get
the current graph with git pull, no rebuild.

Our workaround: a deterministic byte-shard mechanism. A script splits
graph.json into graph.json.part-00, graph.json.part-01, … (95 MiB
chunks — ours are 99,614,720 + 31,139,926 bytes) plus a manifest
(.graphify_shards.json) recording shard order, total bytes, a whole-file
SHA-256, and the commit the graph was built at. A companion script
reconstitutes graph.json by concatenation and verifies the hash. It works,
and resharding is deterministic (identical graph bytes → identical shards, so
git churn is content-proportional).

The failure mode is the consumer side: every tool and every agent has to
know to reconstitute first. When they don't, graphify query (and path/
explain/affected/serve/the MCP server) sees no graph.json and exits
with "graph file not found" — or worse, an agent hook that gates on
graph.json existing silently skips graph usage entirely. In an
agent-driven repo that's a silent no-op failure class: the graph is right
there in the checkout, in pieces, and nothing uses it. We've had CI runs and
agent sessions quietly operate graph-less for exactly this reason.

Proposal

Teach the load path: when the resolved graph path does not exist but
<path>.part-* files and a shard manifest are present in the same directory,
reconstitute transparently (in memory, or to the target path when the size cap
allows) and proceed. One warning line, verified against the manifest SHA-256.

The reason this is cheap: the loaders are already a narrow choke point.
In v8:

• serve.py — _load_graph() (~line 21) is the single loader for the MCP
  server; it already does existence check → check_graph_file_size_cap →
  json.loads → node_link_graph. The shard fallback slots in right at the
  resolved.exists() check (serve.py ~line 26).
• affected.py — load_graph() (~line 230), same shape.
• __main__.py — the inline load sites for query (~2973), path (~3191),
  explain (~3281), tree (~3908), merge-graphs (~4002), export
  (~4244), benchmark (~4377). These all repeat the same
  size-cap + json.loads + node_link_graph sequence — a shared
  load_graph_json(path) helper (which several past issues have edged
  toward) would collapse them and give the shard fallback a single home.
• Default path resolution is already centralized in paths.py
  (default_graph_json()), so "look for parts next to the expected
  graph.json" needs no new configuration surface.

Spec sketch for the manifest (what we use today; happy to adopt whatever
naming you prefer):

{
  "version": 1,
  "shards": ["graph.json.part-00", "graph.json.part-01"],
  "shard_count": 2,
  "total_bytes": 130754646,
  "sha256": "<whole-file sha256>",
  "built_at_commit": "<git sha the graph was built at>"
}

built_at_commit is worth standardizing while you're here: it lets CI answer
"did anything change since the graph was built?" with one git diff --name-only <built_at>..HEAD — that pre-gate is what keeps our post-merge
update job from spawning a metered LLM run on no-delta merges.

Notes:

• Concatenation only — no format change. graph.json stays graphify's native
  node-link JSON; shards are byte slices of it.
• check_graph_file_size_cap applies to total_bytes before reconstitution,
  so the memory-bomb protection is preserved (our 125 MB graph is comfortably
  under the 512 MiB default cap).
• Writers (export.to_json etc.) need no change for the read side to be
  useful; a graphify shard writer subcommand would be a natural follow-up
  but isn't required.

Why not just tell consumers to reconstitute?

We do — and it keeps failing, because "consumers" includes every hook, CI
wrapper, teammate clone, and agent session, and because the failure is silent
(the tools are correct to say the file is missing). A loader that
understands its own committed representation removes the whole class.

Related

• [RFC] Native graphify update for the semantic layer — a scripted conductor with a pluggable LLM hook (we measured: the math is 0.89s–52s, the agent-walked workaround is 11–14 min) #1710 (native update conductor) — its first step is exactly this
  reconstitution; with a native shard loader that step disappears.
• [Bug] build_merge severs edges from UNCHANGED files into re-extracted files (−12% doc↔code in one update cycle): endpoint nodes are replaced, the surviving edges dangle, and the dangling-edge drop silently eats them #1711 (build_merge edge preservation) — same committed-graph
  workflow, merge-step correctness.
• Stamp graph.json with generated_at, warn on stale graphify query #1624 (stamp graph.json with generated_at / warn on stale query) — same
  spirit: the graph artifact should carry the metadata its consumers need.

Offer

We have a working shard/reconstitute implementation (deterministic split,
SHA-256 verify, built_at_commit extraction that scans the stream rather than
JSON-parsing 125 MB) running in CI today. Happy to contribute the PR —
loader fallback + shared load_graph_json helper + tests (round-trip,
truncated-shard, hash-mismatch, size-cap) — if the shape works for you.