diff --git a/propose/active/INDEX-AUTO-MODE-PROPOSE.md b/propose/active/INDEX-AUTO-MODE-PROPOSE.md new file mode 100644 index 00000000..9a33cd6c --- /dev/null +++ b/propose/active/INDEX-AUTO-MODE-PROPOSE.md @@ -0,0 +1,187 @@ +# Proposal: Smart `increment` / `reprocess` Mode Selection + +Status: **active — ready for planning**. +Companion proposal: [`propose/active/TIER2-INCREMENTAL-REBUILD-PROPOSE.md`](TIER2-INCREMENTAL-REBUILD-PROPOSE.md) +(Kuzu incremental rebuild implementation). + +## Goal + +Extend `java-codebase-rag increment` so it can choose incremental vs full +rebuild automatically for both Lance and Kuzu, while staying safe for +rename/delete/move and indexing-semantics changes. + +## Problem + +Current behavior: + +- `java-codebase-rag increment` updates Lance vectors incrementally + (via CocoIndex) but prints a warning that the Kuzu graph is stale. +- `java-codebase-rag reprocess` always does a full rebuild of both + Lance and Kuzu. + +The `increment` command is correct for Lance but incomplete for Kuzu. +The Kuzu incremental path is defined in +[`TIER2-INCREMENTAL-REBUILD-PROPOSE.md`](TIER2-INCREMENTAL-REBUILD-PROPOSE.md). +This proposal defines the decision engine that both commands use to +determine when incremental is safe. + +## Proposed API Changes + +### CLI + +No new commands. Existing commands gain automatic mode selection: + +- `java-codebase-rag increment` — incremental Lance + Kuzu when safe, + full Kuzu fallback when not. Lance is always incremental via CocoIndex. +- `java-codebase-rag reprocess` — full rebuild of both Lance + Kuzu + (unchanged). +- `java-codebase-rag reprocess --graph-only` — full Kuzu rebuild only. +- `java-codebase-rag reprocess --vectors-only` — full Lance rebuild only. + +### MCP tool (`refresh_code_index`) + +Add optional inputs: + +- `confirm: bool = false` (existing) +- `mode: "auto" | "incremental" | "full" = "auto"` +- `changed_paths: list[str] | null = null` +- `git_ref_base: str = "HEAD"` +- `reason: str | null = null` + +Backward compatibility: + +- Calls passing only `confirm=true` should still work. +- Default mode is `auto` (safe-by-default decisioning). + +## Decision Engine (`mode=auto`) + +Returns two independent mode choices — Lance and Kuzu may use different +modes: + +```python +@dataclass +class RefreshDecision: + lance_mode: Literal["incremental", "full"] + kuzu_mode: Literal["incremental", "full"] + reasons: list[str] + detected_changes: ChangeSet +``` + +### Choose `full` for Kuzu when any of the following is true + +- At least one file is deleted. +- At least one file is renamed or moved. +- `.java-codebase-rag.yml` or `.lancedb-mcp.yml` changes. +- Indexing pipeline/config files change (for example: + `java_index_flow_lancedb.py`, `build_ast_graph.py`, + `graph_enrich.py`, enrichment/chunking components). +- `@interface` definitions changed (meta-annotation fanout risk). +- `.deps.json` is missing, corrupt, or has wrong `ontology_version`. +- Change detection fails or is ambiguous. +- More than 50% of files are dirty (incremental would be slower). + +### Choose `full` for Lance when any of the following is true + +- Same config/indexing-pipeline triggers as Kuzu. +- `.java-codebase-rag.yml` changes. +- CocoIndex flow definition changes. + +### Choose `incremental` when all are true + +- Only in-place file content modifications/additions. +- No rename/delete/move events. +- No config/index/meta-annotation risk triggers. +- `.deps.json` exists and is current (Kuzu incremental prerequisite). + +## Change Detection Strategy + +1. Prefer git diff status: + - `git diff --name-status ...HEAD` + - optionally include working tree status (`git diff --name-status` + and `git diff --name-status --cached`) +2. If git signal is unavailable, use `changed_paths` when supplied. +3. If still uncertain, fall back to `full`. + +Represent results as: + +- `added[]`, `modified[]`, `deleted[]`, `renamed[]` +- plus boolean risk flags for config/index/meta-annotation changes + +## Execution Plan + +### Full mode + +- Lance: `cocoindex update ... --full-reprocess -f` +- Kuzu: `build_ast_graph.py --source-root ...` (full rebuild via + `_drop_all`) + +### Incremental mode + +- Lance: `cocoindex update ... -f` (without `--full-reprocess`) +- Kuzu: `build_ast_graph.py --source-root ... --changed-paths ...` + (incremental rebuild via TIER2 proposal) + +The decision engine returns two independent mode choices — Lance and +Kuzu may incrementally update independently. For example, if +`.deps.json` is missing but no config changed, Lance could be +incremental while Kuzu falls back to full. + +## Response Payload Enhancements + +Add decision transparency fields to `refresh_code_index` response: + +- `effective_mode: { lance: "incremental" | "full", kuzu: "incremental" | "full" }` +- `decision_reasons: list[str]` +- `detected_changes: { added, modified, deleted, renamed }` +- optional `warnings: list[str]` + +Keep existing stdout/stderr/exit_code fields intact. + +For the CLI (`increment` command), emit the mode decision to stderr +with `[graph]` / `[vectors]` prefixes consistent with existing +progress output (see `CLI-PROGRESS-OUTPUT-PROPOSE.md`). + +## Safety Policy + +- Default `auto`. +- On uncertainty, choose `full`. +- `mode=full` always respected. +- `mode=incremental` allowed, but return warnings when risk triggers + exist. +- Never silently downgrade explicit `full` to incremental. +- Kuzu incremental failure at runtime → roll back, fall back to full + Kuzu rebuild, log reason. + +## Test Plan + +1. `auto` + modified-only → incremental (both Lance + Kuzu). +2. `auto` + deleted file → full Kuzu, incremental Lance. +3. `auto` + renamed file → full Kuzu, incremental Lance. +4. `auto` + `.java-codebase-rag.yml` change → full (both). +5. `auto` + detection failure → full with reason. +6. explicit `mode=full` → full regardless of diffs. +7. explicit `mode=incremental` + risky changes → incremental + warning. +8. backward compatibility: `confirm=true` only call still succeeds. +9. `.deps.json` missing → full Kuzu, incremental Lance. +10. `.deps.json` stale `ontology_version` → full Kuzu, incremental Lance. + +## Implementation Notes + +- Keep mode-selection logic isolated in helper functions for testability: + - `_detect_repo_changes(...)` + - `_choose_refresh_mode(...)` +- Make logs/user-facing messages explicit about why full mode was + selected. +- Preserve current subprocess environment and project-root behavior. +- The decision engine lives in a shared module (e.g. `index_common.py` + or a new `refresh_decision.py`) usable by both `cli.py` and + `server.py`. + +## Rollout + +1. Implement decision engine with isolated helpers + tests. +2. Integrate into `cli.py`'s `_cmd_increment` — remove + `_emit_increment_kuzu_warning()`, dispatch to Kuzu incremental or + full based on decision. +3. Integrate into `server.py`'s `refresh_code_index` MCP tool. +4. Update `README.md` and `docs/JAVA-CODEBASE-RAG-CLI.md`. diff --git a/propose/active/TIER2-INCREMENTAL-REBUILD-PROPOSE.md b/propose/active/TIER2-INCREMENTAL-REBUILD-PROPOSE.md new file mode 100644 index 00000000..51548eb3 --- /dev/null +++ b/propose/active/TIER2-INCREMENTAL-REBUILD-PROPOSE.md @@ -0,0 +1,560 @@ +# Tier 2 — Incremental Kuzu rebuild + +Status: **active — ready for planning**. +User-facing tracking: GitHub issue **#73**. +Companion proposal: [`propose/active/INDEX-AUTO-MODE-PROPOSE.md`](INDEX-AUTO-MODE-PROPOSE.md) +(decision engine for `increment`/`reprocess`). + +This is a **proposal**, not an implementable plan. After review, +an implementable `plans/active/PLAN-TIER2-INCREMENTAL-REBUILD.md` will +be derived. + +## TL;DR + +`build_ast_graph.py` is a full rebuild on every run: `write_kuzu` calls +`_drop_all` then writes all nodes and edges from scratch. Every `.java` +file is re-parsed and every node/edge re-derived. That worked through +Tier 1B because every fixture is small and users run `reprocess` manually +after big changes. As the tool reaches more users running +`java-codebase-rag increment` frequently, the full-rebuild cost becomes +the dominant latency. This proposal introduces a **diff-driven Kuzu +rebuild path** that updates only what's changed at file granularity, +while keeping the full-rebuild path as the safe fallback. + +The diff-driven path is gated by the decision engine defined in +[`INDEX-AUTO-MODE-PROPOSE.md`](INDEX-AUTO-MODE-PROPOSE.md) — meaning +incremental is *only* used when the decision engine decides it's safe. +Full rebuild remains the default for renames, deletes, config changes, +and ambiguous cases. + +## Why now + +1. **The schema is stable.** Ontology 16 (with Client, Producer, + OVERRIDES, UnresolvedCallSite) is the surface that downstream + tooling depends on. Adding incremental rebuild now doesn't risk + schema churn. +2. **The decision engine is co-proposed.** `INDEX-AUTO-MODE-PROPOSE.md` + specifies when full vs incremental is safe for both Lance and Kuzu. +3. **Users need it.** The `increment` command updates Lance vectors + but prints a warning that the Kuzu graph is stale. Users running + edit-index-query loops get divergent results. +4. **Idempotency hooks are in place.** `pass6_match_edges` resets + its match-breakdown counters at the start of every run. Future + passes can adopt the same contract trivially. +5. **Single-developer pain point.** Edit-rebuild-query loops are the + inner loop. A sub-second incremental rebuild is the difference + between the tool feeling responsive and feeling batch-oriented. + +## Goals + +1. **File-level incremental rebuild** that touches only nodes/edges + derived from changed `.java` files (and their dependency closure — + see §2.3). +2. **Bit-for-bit equivalence** between incremental and full rebuild for + the same final source-tree state (verified by determinism test). +3. **Safe fallback to full rebuild** in any ambiguous case, never + silent partial state. +4. **No schema churn** — the surface (`graph_meta`, ontology 16, MCP + tools) is identical; incremental is a build-strategy optimisation. +5. **CLI integration via `increment`** — `java-codebase-rag increment` + updates both Lance and Kuzu incrementally. `reprocess` remains the + explicit full-rebuild command. `build_ast_graph.py` gains + `--changed-paths` as an internal flag (not user-facing). + +## Non-goals + +- Watch-mode (filesystem watcher running continuously). + This proposal lays the foundation; a future + `propose/WATCH-MODE-PROPOSE.md` can build on top. +- Multi-tenant/concurrent rebuilds. Single writer assumed. +- Distributed/sharded indexing. Out of scope. +- LanceDB incremental rebuild (already covered by CocoIndex's native + incremental in the `increment` command — this is Kuzu-only). +- Cross-repo or cross-source-root incremental. One source tree at a time. +- Schema migrations between ontology versions. Schema bumps still + require a full rebuild — flagged by the decision engine. + +## 1. Current state + +### 1.1 The full-rebuild pipeline + +``` +build_ast_graph.main() + └─ pass1_parse(root, tables) # parse every .java → JavaFileAst + └─ pass2_edges(tables, asts) # EXTENDS/IMPLEMENTS/INJECTS edges + └─ pass3_calls(tables, asts) # CALLS + UnresolvedCallSite edges + └─ pass4_routes(tables, asts) # Route extraction (4 strategies) + └─ pass5_imperative_edges(tables, asts) # Client/Producer nodes, HTTP_CALLS/ASYNC_CALLS + └─ pass6_match_edges(tables) # match resolution + cross-service + └─ write_kuzu(kuzu_path, tables) # _drop_all → _create_schema → write nodes/edges/meta + └─ _drop_all(conn) # all tables dropped + └─ _create_schema(conn) # recreate from scratch + └─ _write_nodes(conn, tables) # Symbol, Route, Client, Producer nodes + └─ _populate_declares_rows # DECLARES edges + └─ _populate_overrides_rows # OVERRIDES edges (subtype→supertype method) + └─ _write_edges(conn, tables) # EXTENDS/IMPLEMENTS/INJECTS/CALLS/OVERRIDES + └─ _write_routes_and_exposes # Route/EXPOSES, Client/DECLARES_CLIENT, Producer/DECLARES_PRODUCER + └─ _write_meta(conn, tables) # graph_meta JSON-blob columns +``` + +Every pass populates the in-memory `GraphTables` struct. `write_kuzu` +then drops the entire Kuzu DB and writes everything from scratch. + +### 1.2 What's already well-positioned for incremental + +- **Pass1 is per-file** — each `JavaFileAst` is independent. Parsing is + trivially incrementalisable. +- **Pass6 is idempotent** — match counters reset, then re-derive from + the edges in the DB. With pass5 edges intact for unchanged files, + `pass6` can re-run cheaply over the union of "edges from changed + files" + "all matched edges that *might* now have new candidates". +- **Pass4's route extraction** is per-file at extraction time but + cross-file at resolution time (3 strategies). Route hints from one + file can affect another file's resolution. +- **`GraphTables` is an in-memory accumulator** — passes add to it, + `write_kuzu` reads from it. An incremental path can construct a + partial `GraphTables` (only dirty files' data) and merge it into the + existing DB. + +### 1.3 What blocks naive incremental + +- **Pass2's edges (EXTENDS/IMPLEMENTS/INJECTS)** are computed across + files. If `Foo.java` extends `Bar.java`, both are needed to emit the + edge. Editing `Bar.java` invalidates the edge from `Foo`. +- **Pass3's CALLS edges** depend on resolution state across files + (a method call in `A.java` may resolve to a method in `B.java`). + Pass3 also emits `UnresolvedCallSite` + `UNRESOLVED_AT` edges. +- **OVERRIDES edges** span files — `Foo.method()` overrides + `Bar.method()`. Editing `Bar.java` invalidates the OVERRIDES edge. + These are computed in `_populate_overrides_rows` during the write + phase. +- **Brownfield Layer 4 + 5** (`@CodebaseClient`, `@CodebaseProducer`, + meta-annotation chains) propagate effects: editing the meta-annotation + hub silently changes every annotated method downstream. +- **Pass5 emits Client/Producer nodes** in addition to HTTP_CALLS/ + ASYNC_CALLS edges. These nodes have their own DECLARES_CLIENT/ + DECLARES_PRODUCER edges. HTTP_CALLS now goes Client→Route and + ASYNC_CALLS goes Producer→Route (not Symbol→Route as in ontology 7). +- **Pass6 cross-service matching** matches caller-side calls against + routes from *all* services. A new route in svc-b can flip an + unmatched HTTP_CALLS edge in svc-a to `cross_service`. + +These constraints mean we need **change-set closure**, not just +file-level dirty tracking. + +## 2. Design + +### 2.1 Two-tier change set + +The runtime computes a **change set** with two tiers: + +- **Tier 1 (direct):** files in `--changed-paths` (added, modified). +- **Tier 2 (closure):** files reachable via "depends on" links from + Tier 1 — see §2.3. + +Both tiers are re-parsed and their nodes/edges deleted from the DB +before pass1's incremental write. Files outside both tiers are left +untouched. + +### 2.2 The incremental main loop (sketch) + +``` +build_ast_graph_incremental(changed_paths): + if decision_engine.requires_full(changed_paths): + return run_full_rebuild() # safe fallback + + # Compute change set (Tier 1 + Tier 2 closure) + dirty = expand_to_closure(changed_paths) + + with conn: + delete_rows_for_files(conn, dirty) # symmetric deletes (see §2.5) + asts = pass1_parse_subset(root, dirty) # only re-parse dirty files + tables = GraphTables() # fresh accumulator — only dirty data + pass2_edges_subset(tables, asts, dirty) # re-emit cross-file edges + # for symbols touching dirty + pass3_calls_subset(tables, asts, dirty) + pass4_routes_subset(tables, asts, dirty) + pass5_imperative_edges_subset(tables, asts, dirty) + + # OVERRIDES for dirty files only + _populate_overrides_rows(tables) + + # Pass6 always reruns globally — its cost is proportional to total + # call edges, not changed files, and it computes match outcomes + # that span services. Cheap (~ms on bank-chat-system) and worth + # the simplicity. See §3.6. + pass6_match_edges(tables) + + # Write incremental rows to existing DB (no _drop_all) + _write_nodes_incremental(conn, tables) + _write_edges_incremental(conn, tables) + _write_routes_and_exposes_incremental(conn, tables) + _write_meta(conn, tables, source_root) # always rewritten + _write_dependency_index(deps_path, asts) # sidecar .deps.json (see §2.4) +``` + +### 2.3 Change-set closure rules + +For each file `F` in Tier 1, mark these files as Tier 2: + +1. **Inverse-INJECTS:** every file that `INJECTS` a symbol declared in `F`. +2. **Inverse-EXTENDS / Inverse-IMPLEMENTS:** every file whose symbol + extends or implements a symbol from `F`. (Class hierarchy + re-resolution may change brownfield-overrides resolution.) +3. **Inverse-CALLS:** every file that calls a symbol declared in `F`. + (CALLS resolution can flip targets when overload sets change.) +4. **Meta-annotation closure:** if `F` declares an `@interface` (a + meta-annotation), every file that uses that annotation is dirty. + (Brownfield Layer 5 can re-fanout.) +5. **Brownfield-override closure:** if `F` contains + `@CodebaseClient` / `@CodebaseProducer` / `@CodebaseEndpoint` / + role/capability override annotations, every file in the same + FQN scope (per Layer 4 rules) is potentially affected — fall + back to full rebuild (see TBD-1 resolution below). +6. **Route resolution closure:** if `F` contains route declarations + or `@RequestMapping`-class hints, every file with a method on + that class is dirty. +7. **Inverse-OVERRIDES:** if `F` declares a method that overrides a + supertype method, editing the supertype's file invalidates the + subtype's OVERRIDES edge — the subtype's file is dirty. +8. **Inverse-DECLARES_CLIENT / Inverse-DECLARES_PRODUCER:** if `F` + declares a Client or Producer, and a route change in another file + could affect the match outcome, the Client/Producer's file is + dirty. In practice, pass6 already reruns globally and handles + match resolution — this closure ensures the Client/Producer + *nodes* are current when their declaring method's file changes. + +The `_write_dependency_index` step (§2.4) caches the inverse maps so +closure is O(|dirty| × avg_inverse_degree), not O(|dirty| × |all-files|). + +### 2.4 Dependency index + +A sidecar file at `/.deps.json` stores per-file dependency +metadata. This is a build cache, not graph data — kept outside the Kuzu +DB for cheap Python-level inverse-map computation and no Cypher overhead. + +```json +{ + "version": 1, + "ontology_version": 16, + "files": { + "src/main/java/com/example/Foo.java": { + "ext_hash": "sha256:abc...", + "declares": ["com.example.Foo", "com.example.Foo.Bar"], + "injects": ["com.example.Service"], + "extends": ["com.example.Base"], + "calls": ["com.example.Service#run"], + "uses_anno": ["@RestController", "@CodebaseEndpoint"], + "overrides": ["com.example.Base#method()"], + "declares_clients": ["com.example.Foo#callApi()"], + "declares_producers": [] + } + } +} +``` + +Atomicity via write-temp-rename pattern. Version field for future +migrations. `ontology_version` field to detect stale index — mismatch +triggers full rebuild. + +### 2.5 Symmetric delete + +Every pass that *emits* nodes/edges keyed by source file must have a +symmetric `delete_*_for_file(path)` helper. Concretely: + +| Pass | Emits | Delete helper | +|------|-------|---------------| +| 1 | `Symbol`, `DECLARES` | `delete_symbols_for_file(p)` (cascades to DECLARES) | +| 2 | `EXTENDS`, `IMPLEMENTS`, `INJECTS` | per-edge filter on src-symbol's file | +| 3 | `CALLS`, `UnresolvedCallSite`, `UNRESOLVED_AT` | per-edge/site filter on caller-symbol's file | +| 4 | `Route`, `EXPOSES` | per-route filter on owner-symbol's file | +| 5 | `Client`, `DECLARES_CLIENT`, `Producer`, `DECLARES_PRODUCER` | per-node filter on declaring-method's file | +| 5 | `HTTP_CALLS` (Client→Route) | per-edge filter on Client's declaring-method's file | +| 5 | `ASYNC_CALLS` (Producer→Route) | per-edge filter on Producer's declaring-method's file | +| overrides | `OVERRIDES` | per-edge filter on subtype-method's file | + +Each helper takes a `Path` and a `kuzu.Connection`, executes the +DELETE Cypher, and returns a count for `--verbose` logging. The file +path is reachable via the source `Symbol`'s `path` field (or via +`Client.filename` / `Producer.filename` for those node types). + +### 2.6 Change-detection sources + +Three sources, in priority order: + +1. **`--changed-paths` flag** (or stdin list). Caller responsibility + to be accurate. This is what the INDEX-AUTO-MODE decision engine + would pass. +2. **Git diff** between `--git-ref-base` and `HEAD`. Same logic as + INDEX-AUTO-MODE §"Change Detection Strategy". +3. **Hash-based detection** using `.deps.json` `ext_hash` fields. Walk + the source tree, compute hashes, compare against cached. Slowest; + used as last-resort fallback. + +If all three fail or are ambiguous, fall back to full rebuild. + +## 3. Pass-by-pass incrementalisation notes + +### 3.1 Pass1 (parse) + +Trivial. Re-parse only files in the dirty set. No global state. + +### 3.2 Pass2 (cross-file edges) + +The edge `Foo EXTENDS Bar` lives in `pass2`. If only `Foo.java` +changed, the edge needs to be re-emitted with up-to-date metadata. +If only `Bar.java` changed, the edge target is unchanged but `Bar`'s +symbol ID may have new fields — re-emit. + +OVERRIDES edges (computed in `_populate_overrides_rows` during write) +follow the same pattern. If `Foo.method()` overrides `Bar.method()` +and `Bar.java` changes, `Foo`'s OVERRIDES edge is stale — covered by +closure rule 7. + +### 3.3 Pass3 (CALLS + UnresolvedCallSite) + +Same shape as pass2. CALLS edges re-emitted for caller-symbols in +dirty files. Closure rule 3 catches inverse-CALLS. `UnresolvedCallSite` +rows and their `UNRESOLVED_AT` edges are also deleted and re-emitted +for the dirty subset. + +### 3.4 Pass4 (Routes) + +Two sub-passes: + +- **Extraction** is per-file. Easy. +- **Resolution** (three-strategy ladder) reads class-level hints from + a possibly-different file. Closure rule 6 covers this. + +Phantom-route cleanup currently runs in `pass6`. That stays as-is — +phantoms are derived from the global graph state. + +### 3.5 Pass5 (Client/Producer + HTTP_CALLS/ASYNC_CALLS) + +Now emits Client and Producer *nodes* (not just edges) plus their +DECLARES_CLIENT/DECLARES_PRODUCER edges. HTTP_CALLS now originates from +Client→Route and ASYNC_CALLS from Producer→Route. + +Symmetric delete must clean up all of these per-file. Brownfield +Layer 4 (`@CodebaseClient` / `@CodebaseProducer`) overrides — closure +rule 5 currently forces full rebuild when an override file changes; +pessimistic but safe. Fine-grained brownfield closure deferred. + +### 3.6 Pass6 (match outcomes + cross-service) + +**Always reruns globally.** Three reasons: + +1. Cross-service matching can flip outcomes service-wide when one + route's path template changes. +2. Pass6 is fast: on `bank-chat-system` it runs in ~1ms. Even at 10× + scale we're still under the human-perception threshold. +3. Pass6's idempotency contract was already added with incremental + in mind. + +### 3.7 `_write_meta` + +Always rewritten — fields like `routes_total`, `cross_service_calls_total`, +`http_calls_match_breakdown`, `async_calls_match_breakdown` are aggregations +over the global graph state. Cost is negligible. + +## 4. CLI surface + +The incremental path integrates with the existing `java-codebase-rag` +CLI, not as a user-facing flag on `build_ast_graph.py`: + +```bash +# Existing — unchanged (full rebuild of Lance + Kuzu) +java-codebase-rag reprocess +java-codebase-rag reprocess --graph-only +java-codebase-rag reprocess --vectors-only + +# Updated — now also increments Kuzu graph (not just Lance) +java-codebase-rag increment + +# Internal — build_ast_graph.py gains --changed-paths (called by CLI, not users) +python build_ast_graph.py \ + --source-root --kuzu-path \ + --changed-paths \ + [--verbose] +``` + +Decision engine inside `cli.py`'s `_cmd_increment`: + +1. If `` doesn't exist or `.deps.json` is missing/stale → + fall back to full graph rebuild. +2. If `decision_engine.requires_full(changed)` → full graph rebuild + with a logged reason. +3. If incremental fails mid-flight → roll back transaction, log, + fall back to full graph rebuild. +4. On success → both Lance and Kuzu are incremental. + +The `_emit_increment_kuzu_warning()` call in `cli.py` is removed once +this ships. + +## 5. MCP integration + +`refresh_code_index` (MCP tool in `server.py`) gains the ability to +call the incremental Kuzu path via the CLI: + +```python +# Pseudocode inside server.py:refresh_code_index +decision = decide_mode(changed_paths, git_ref_base) +if decision.lance_mode == "incremental": + cocoindex_update(incremental=True, paths=decision.lance_paths) +else: + cocoindex_update(full=True) + +if decision.kuzu_mode == "incremental": + subprocess.run(["java-codebase-rag", "increment"]) +else: + subprocess.run(["java-codebase-rag", "reprocess", "--graph-only"]) +``` + +Decision engine returns two independent mode choices — LanceDB and +Kuzu may incrementally update independently. See +[`INDEX-AUTO-MODE-PROPOSE.md`](INDEX-AUTO-MODE-PROPOSE.md) for the +full decision engine specification. + +## 6. Determinism + correctness + +### 6.1 Determinism test + +Create `tests/test_incremental_equivalence.py` as a prerequisite: + +```python +def test_incremental_matches_full(tmp_path): + src = tmp_path / "src"; src.mkdir() + copy_fixture(src, "tests/fixtures/cross_service_smoke") + + # Full rebuild on initial state + run_full(src, kuzu_a := tmp_path / "a") + + # Mutate one file + (src / "svc-a/.../ClientA.java").write_text(modified_source) + + # Two paths to the same final state + run_full(src, kuzu_full := tmp_path / "full") + run_incremental(kuzu_a, changed=["svc-a/.../ClientA.java"]) + + assert all_sorted_ids(kuzu_full) == all_sorted_ids(kuzu_a) + assert meta(kuzu_full) == meta(kuzu_a) +``` + +This is the single most important test in the entire feature. +Without it, every incremental rebuild is a potential silent-divergence +risk. Run against every fixture. + +### 6.2 Failure modes + +- **Half-applied delete:** transaction wraps the entire incremental + pass; on any exception, ROLLBACK and fall back to full rebuild. +- **Corrupt `.deps.json`:** detected by checksum + `ontology_version` + field; treated as "incremental impossible, fall back to full". +- **Schema-version mismatch:** if `graph_meta.ontology_version` ≠ + current `ONTOLOGY_VERSION`, force full rebuild. + +## 7. Rollout + +1. **PR-T1 (foundation + determinism test):** Create + `tests/test_incremental_equivalence.py` with full-rebuild + determinism assertions. Add `.deps.json` writer + (`_write_dependency_index`) and reader, written but unused. + Establish performance baseline on fixtures. +2. **PR-T2 (delete helpers):** symmetric `delete_*_for_file` for each + pass and node/edge type (Symbol, DECLARES, EXTENDS, IMPLEMENTS, + INJECTS, CALLS, UnresolvedCallSite, UNRESOLVED_AT, Route, EXPOSES, + Client, DECLARES_CLIENT, Producer, DECLARES_PRODUCER, HTTP_CALLS, + ASYNC_CALLS, OVERRIDES). Unit-tested in isolation. Still no + incremental codepath. +3. **PR-T3 (incremental orchestrator):** `build_ast_graph_incremental` + function + `--changed-paths` flag on `build_ast_graph.py`. Pass-by- + pass incremental implementation including Client/Producer/OVERRIDES/ + UnresolvedCallSite. `test_incremental_matches_full` extended to + cover all fixtures. +4. **PR-T4 (CLI + decision engine):** Integrate INDEX-AUTO-MODE + decision engine into `cli.py`'s `_cmd_increment`. Remove + `_emit_increment_kuzu_warning()`. Wire `refresh_code_index` + MCP tool to dispatch. +5. **PR-T5 (brownfield closure refinement, optional):** Narrow the + pessimistic "any brownfield-override change → full" rule once + Layer-4/5 fanout is explicitly documented. + +PR-T1 through PR-T4 are the headline. PR-T5 is an optimisation +on top. + +## 8. Risk assessment + +| Risk | Severity | Mitigation | +|------|----------|------------| +| Silent divergence between full and incremental | **High** | `test_incremental_matches_full` mandatory per fixture; covered by §6.1 | +| Pessimistic full-fallback hides real incremental wins | Medium | Track full-fallback rate via `graph_meta.last_rebuild_mode` + reason; review monthly | +| `.deps.json` becomes stale | Medium | `ontology_version` field check; mismatch → full rebuild | +| Decision engine bugs cause unsafe incremental | High | Default-to-full on ambiguity; conservative initial rules; expand only after burn-in | +| Pass2/Pass3 closure misses an edge type | Medium | Determinism test surfaces this immediately | +| Performance regression (incremental slower than full for small repos) | Low | Heuristic: skip incremental when dirty set is >50% of files | + +## 9. Performance estimate + +On a hypothetical 1000-file Java repo: + +| Metric | Full rebuild | Incremental (1 file changed) | +|--------|--------------|-------------------------------| +| Pass1 parse | ~30s (1000 × 30ms) | <0.1s (1 × 30ms + 5-10 closure files) | +| Pass2-5 emit | ~20s | <0.2s | +| Pass6 match | ~1s | ~1s (always global) | +| `_write_meta` | <0.1s | <0.1s | +| **Total** | **~50s** | **~1.5s** | + +These are estimates — actual numbers depend on closure breadth and +Kuzu transaction overhead. PR-T1 establishes a measured baseline on +`bank-chat-system` and other fixtures. + +## 10. Resolved TBDs + +### TBD-1: Brownfield closure granularity — **pessimistic fallback** + +Layer-4 (`@CodebaseClient` / `@CodebaseProducer`) overrides operate +via FQN matching. Editing the override file conceptually invalidates +every matching method. The current proposal forces full rebuild in +that case (closure rule 5). Fine-grained closure is deferred to PR-T5 +or a follow-up proposal once Layer-4 fanout rules are formalised. + +### TBD-2: FileDeps storage — **sidecar JSON** + +Decision: **sidecar `/.deps.json`**. The file-level dependency +index is conceptually a build cache, not graph data. Benefits: cheap +inverse-map computation in Python, no Cypher overhead, no schema +evolution burden. Atomicity via write-temp-rename. `version` + +`ontology_version` fields for migration/staleness detection. + +### TBD-3: Incremental for the LanceDB chunks — **out of scope** + +Already covered by CocoIndex's native incremental support +(`java-codebase-rag increment` without `--full-reprocess`). + +### TBD-4: Concurrent / interleaved rebuilds — **document only** + +Single-writer assumed. If two runs race against the same `--kuzu-path`, +behaviour is undefined. Document; do not guard against. + +### TBD-5: Watch-mode — **out of scope, future proposal** + +A follow-on proposal should layer a filesystem watcher on top of this +incremental path. It needs debouncing, batched-change semantics, and +its own state machine. + +## 11. Done definition (proposal-level) + +This proposal is "ready for plan derivation" when: + +- [x] All TBD items have a decision. +- [ ] §3 closure rules have been validated against a real + multi-service fixture (`cross_service_smoke` is a candidate). +- [ ] §9 performance estimates have a measured baseline on at least + one fixture. +- [ ] Reviewer approves §2.5 (symmetric delete) as the right model. +- [ ] Decision engine pseudocode in §4 + §5 is consistent with + `INDEX-AUTO-MODE-PROPOSE.md`. + +When approved, derive `plans/active/PLAN-TIER2-INCREMENTAL-REBUILD.md` +with PR-T1 through PR-T5 broken out, and a per-PR agent task prompt set. diff --git a/propose/stale/INDEX-AUTO-MODE-PROPOSE.md b/propose/stale/INDEX-AUTO-MODE-PROPOSE.md deleted file mode 100644 index 0caa6a45..00000000 --- a/propose/stale/INDEX-AUTO-MODE-PROPOSE.md +++ /dev/null @@ -1,124 +0,0 @@ -# Proposal: Smart `refresh_code_index` Mode Selection - -> **Note (2026):** the operator CLI’s full rebuild verb is now `java-codebase-rag reprocess` (see `propose/completed/CLI-SCENARIOS-PROPOSE.md`); this MCP-side proposal still refers to `refresh_code_index` by name. -> -> For embeddings or search over raw `propose/` text, expect both that historical MCP name and current CLI wording in the same document until this draft is implemented or archived under `propose/completed/`. - -## Goal - -Extend `refresh_code_index` so it can choose incremental vs full rebuild automatically, while staying safe for rename/delete/move and indexing-semantics changes. - -## Problem - -Current behavior always runs: - -- `cocoindex update ... --full-reprocess -f` -- followed by full Kuzu rebuild - -This is correct but expensive for small edit-only iterations. At the same time, a naive incremental flow can leave stale entries after file deletes/renames or when metadata semantics change. - -## Proposed API Changes - -Add optional inputs to `refresh_code_index`: - -- `confirm: bool = false` (existing) -- `mode: "auto" | "incremental" | "full" = "auto"` -- `changed_paths: list[str] | null = null` -- `git_ref_base: str = "HEAD"` -- `reason: str | null = null` - -Backward compatibility: - -- Calls passing only `confirm=true` should still work. -- Default mode should become `auto` (safe-by-default decisioning). - -## Decision Engine (`mode=auto`) - -### Choose `full` when any of the following is true - -- At least one file is deleted. -- At least one file is renamed or moved. -- `.lancedb-mcp.yml` changes. -- Indexing pipeline/config files change (for example: `java_index_flow_lancedb.py`, enrichment/chunking components). -- `@interface` definitions changed (meta-annotation fanout risk). -- Change detection fails or is ambiguous. - -### Choose `incremental` when all are true - -- Only in-place file content modifications/additions. -- No rename/delete/move events. -- No index/config/meta-annotation risk triggers. - -## Change Detection Strategy - -1. Prefer git diff status: - - `git diff --name-status ...HEAD` - - optionally include working tree status (`git diff --name-status` and `git diff --name-status --cached`) -2. If git signal is unavailable, use `changed_paths` when supplied. -3. If still uncertain, fall back to `full`. - -Represent results as: - -- `added[]`, `modified[]`, `deleted[]`, `renamed[]` -- plus boolean risk flags for config/index/meta-annotation changes - -## Execution Plan - -### Full mode - -- Keep current behavior: - - `cocoindex update ... --full-reprocess -f` - - run `build_ast_graph.py --source-root ...` - -### Incremental mode - -- Run `cocoindex update ... -f` (without `--full-reprocess`) -- Graph handling in two phases: - - Phase A: still run full graph rebuild for correctness. - - Phase B (future): incremental graph updates scoped to changed files. - -## Response Payload Enhancements - -Add decision transparency fields: - -- `effective_mode: "incremental" | "full"` -- `decision_reasons: list[str]` -- `detected_changes: { added, modified, deleted, renamed }` -- optional `warnings: list[str]` (for forced incremental under risky conditions) - -Keep existing stdout/stderr/exit_code fields intact. - -## Safety Policy - -- Default `auto`. -- On uncertainty, choose `full`. -- `mode=full` always respected. -- `mode=incremental` allowed, but return warnings when risk triggers exist. -- Never silently downgrade explicit `full` to incremental. - -## Test Plan - -Add/extend tests in `tests/test_mcp_tools.py`: - -1. `auto` + modified-only -> incremental chosen. -2. `auto` + deleted file -> full chosen. -3. `auto` + renamed file -> full chosen. -4. `auto` + `.lancedb-mcp.yml` change -> full chosen. -5. `auto` + detection failure -> full chosen with reason. -6. explicit `mode=full` -> full regardless of diffs. -7. explicit `mode=incremental` + risky changes -> incremental + warning. -8. backward compatibility: `confirm=true` only call still succeeds. - -## Implementation Notes - -- Keep mode-selection logic isolated in helper functions for testability: - - `_detect_repo_changes(...)` - - `_choose_refresh_mode(...)` -- Make logs/user-facing messages explicit about why full mode was selected. -- Preserve current subprocess environment and project-root behavior. - -## Rollout - -1. Implement API and auto decisioning with Phase A graph behavior. -2. Add test coverage and docs update in `README.md`. -3. Later: implement true incremental Kuzu graph updates (Phase B). diff --git a/propose/stale/TIER2-INCREMENTAL-REBUILD-PROPOSE.md b/propose/stale/TIER2-INCREMENTAL-REBUILD-PROPOSE.md deleted file mode 100644 index 73101047..00000000 --- a/propose/stale/TIER2-INCREMENTAL-REBUILD-PROPOSE.md +++ /dev/null @@ -1,520 +0,0 @@ -# Tier 2 — Incremental Kuzu rebuild - -Status: **active — ready for planning**. -User-facing tracking for graph-side incremental work: GitHub issue **#73** (linked from the `increment` stderr warning in `java_codebase_rag/cli.py`). -Pairs with the focused MCP-tool proposal -[`propose/INDEX-AUTO-MODE-PROPOSE.md`](INDEX-AUTO-MODE-PROPOSE.md) -(decision engine for `refresh_code_index`) and supersedes its -"future Kuzu work" footnote in [`docs/PRODUCT-VISION.md`](../docs/PRODUCT-VISION.md) §99. - -This is a **proposal**, not an implementable plan. After review and -scoping decisions (the §11 [TBD] list), an implementable -`plans/PLAN-TIER2-INCREMENTAL-REBUILD.md` will be derived. - -## TL;DR - -`build_ast_graph.py` is a full rebuild on every run: `_drop_all` plus -`pass1`–`pass6` re-parse every `.java` file in the source tree and -re-derive every node and edge. That worked through Tier 1B because -every fixture is small and the production user runs an indexing job -manually after big changes. As we move toward continuous indexing -(`refresh_code_index` with `mode=incremental`, future watch-mode, or -post-merge CI hooks), the full-rebuild cost becomes the dominant -latency. This proposal introduces a **diff-driven Kuzu rebuild path** -that updates only what's changed at file granularity, while keeping -the full-rebuild path as the safe fallback. - -The diff-driven path is gated by the same decision engine that the -REFRESH proposal already specifies for LanceDB — meaning incremental -is *only* used when the decision engine decides it's safe. Full -rebuild remains the default for renames, deletes, config changes, and -ambiguous cases. - -## Why now (and not earlier) - -1. **Tier 1B closed the schema.** Ontology 7 (with `pass6_match_edges`) - is the surface that downstream tooling depends on. Adding incremental - rebuild now doesn't risk schema churn. -2. **The decision engine already exists in proposal form.** REFRESH - proposal §"Decision Engine" specifies exactly when full vs incremental - is safe. The current Kuzu path can't take advantage because there is - no incremental option to dispatch to — this proposal closes that gap. -3. **Idempotency hooks are already added.** `pass6_match_edges` resets - its match-breakdown counters at the start of every run (PR-D3 review - obs 3, addressed in PR-E1 follow-up plan). Future passes can adopt - the same idempotency contract trivially. -4. **Single-developer pain point.** As the user maintains real codebases - with the AMA agent, edit-rebuild-query loops are the inner loop. A - sub-second incremental rebuild is the difference between the agent - feeling responsive and feeling batch-oriented. - -## Goals - -1. **File-level incremental rebuild** that touches only nodes/edges - derived from changed `.java` files (and their meta-annotation / - brownfield-overrides closure — see §3.2). -2. **Bit-for-bit equivalence** between incremental and full rebuild for - the same final source-tree state (verified by determinism test). -3. **Safe fallback to full rebuild** in any ambiguous case, never - silent partial state. -4. **No schema churn** — the surface (`graph_meta`, ontology 7, MCP - tools) is identical; incremental is a build-strategy optimisation. -5. **Backwards-compatible CLI** — `python build_ast_graph.py - --source-root … --kuzu-path …` keeps its current full-rebuild - semantics. A new `--changed-paths …` (or equivalent) opt-in gates - the incremental path. - -## Non-goals - -- Watch-mode (filesystem watcher running continuously). - This proposal lays the foundation; a future `propose/WATCH-MODE-PROPOSE.md` - can build on top. -- Multi-tenant/concurrent rebuilds. Single writer assumed. -- Distributed/sharded indexing. Out of scope. -- LanceDB incremental rebuild (already covered by CocoIndex's native - incremental in the REFRESH proposal — this is Kuzu-only). -- Cross-repo or cross-source-root incremental. One source tree at a time. -- Schema migrations between ontology versions. Schema bumps still - require a full rebuild — flagged by the decision engine. - -## 1. Current state - -### 1.1 The full-rebuild pipeline - -``` -build_ast_graph.main() - └─ _drop_all(conn) # 11 tables dropped - └─ pass1_parse(root) # parse every .java → JavaFileAst - └─ pass2_edges # EXTENDS/IMPLEMENTS/INJECTS edges - └─ pass3_calls # CALLS edges (intra-service) - └─ pass4_routes # Route extraction (4 strategies) - └─ pass5_imperative_edges # HTTP_CALLS / ASYNC_CALLS caller edges - └─ pass6_match_edges # 5-outcome match resolution + cross-service - └─ _write_meta # graph_meta JSON-blob columns -``` - -Every pass either holds rows in `GraphTables` or writes them after the -parse phase. Storage is single-file Kuzu DB at `--kuzu-path`. Phase 1 -*always* drops and recreates. - -### 1.2 What's already well-positioned for incremental - -- **Pass1 is per-file** — each `JavaFileAst` is independent. Parsing is - trivially incrementalisable. -- **Pass6 is idempotent** — match counters reset, then re-derive from - edges in the DB. With `pass5` edges intact for unchanged files, - `pass6` can re-run cheaply over the union of "edges from changed - files" + "all matched edges that *might* now have new candidates". -- **Pass4's route extraction** is per-file at extraction time but - cross-file at resolution time (3 strategies in PR-A2). Route hints - from one file can affect another file's resolution. -- **Determinism test (`tests/test_determinism.py`)** exists and - enforces that two runs on the same input produce identical sorted - IDs. This is the single most important property for incremental - rebuild — we'll extend it to `incremental(input) == full(input)`. - -### 1.3 What blocks naive incremental - -- **Pass2's edges (EXTENDS/IMPLEMENTS/INJECTS)** are computed across - files. If `Foo.java` extends `Bar.java`, both are needed to emit the - edge. Editing `Bar.java` invalidates the edge from `Foo`. -- **Pass3's CALLS edges** depend on resolution state across files - (a method call in `A.java` may resolve to a method in `B.java`). -- **Brownfield Layer 4 + 5** (`@CodebaseClient`, `@CodebaseProducer`, - meta-annotation chains) propagate effects: editing the meta-annotation - hub silently changes every annotated method downstream. -- **Pass5/Pass6 cross-service matching** matches caller-side calls - against routes from *all* services. A new route in svc-b can flip - an unmatched HTTP_CALLS edge in svc-a to `cross_service`. - -These constraints mean we need **change-set closure**, not just -file-level dirty tracking. - -## 2. Design - -### 2.1 Two-tier change set - -The runtime computes a **change set** with two tiers: - -- **Tier 1 (direct):** files in `--changed-paths` (added, modified). -- **Tier 2 (closure):** files reachable via "depends on" links from - Tier 1 — see §3.2. - -Both tiers are re-parsed and their nodes/edges deleted from the DB -before pass1's incremental write. Files outside both tiers are left -untouched. - -### 2.2 The incremental main loop (sketch) - -``` -build_ast_graph_incremental(changed_paths): - if decision_engine.requires_full(changed_paths): - return run_full_rebuild() # safe fallback - - # Compute change set (Tier 1 + Tier 2 closure) - dirty = expand_to_closure(changed_paths) - - with conn: - delete_rows_for_files(dirty) # symmetric to pass1's emits - asts = pass1_parse_subset(dirty) # only re-parse dirty files - pass2_edges_subset(asts, dirty) # re-emit cross-file edges - # for symbols touching dirty - pass3_calls_subset(asts, dirty) - pass4_routes_subset(asts, dirty) - pass5_imperative_edges_subset(asts, dirty) - - # Pass6 always reruns globally — its cost is proportional to total - # call edges, not changed files, and it computes match outcomes - # that span services. Cheap (~ms on bank-chat-system) and worth - # the simplicity. See §3.4. - pass6_match_edges() - - _write_meta(conn, tables, source_root) # always rewritten - _write_dependency_index(conn, asts) # NEW (see §2.4) -``` - -### 2.3 Change-set closure rules - -For each file `F` in Tier 1, mark these files as Tier 2: - -1. **Inverse-INJECTS:** every file that `INJECTS` a symbol declared in `F`. -2. **Inverse-EXTENDS / Inverse-IMPLEMENTS:** every file whose symbol - extends or implements a symbol from `F`. (Class hierarchy - re-resolution may change brownfield-overrides resolution.) -3. **Inverse-CALLS:** every file that calls a symbol declared in `F`. - (CALLS resolution can flip targets when overload sets change.) -4. **Meta-annotation closure:** if `F` declares an `@interface` (a - meta-annotation), every file that uses that annotation is dirty. - (Brownfield Layer 5 can re-fanout.) -5. **Brownfield-override closure:** if `F` contains - `@CodebaseClient` / `@CodebaseProducer` / `@CodebaseEndpoint` / - role/capability override annotations, every file in the same - FQN scope (per Layer 4 rules) is potentially affected — fall - back to full rebuild for now (see §11 [TBD-1]). -6. **Route resolution closure:** if `F` contains route declarations - or `@RequestMapping`-class hints, every file with a method on - that class is dirty (PR-A2's three-strategy ladder reads class-level - `@RequestMapping` from a possibly-different file). - -The `_write_dependency_index` step (§2.4) caches the inverse maps so -closure is O(|dirty| × avg_inverse_degree), not O(|dirty| × |all-files|). - -### 2.4 Dependency index - -A new node table `FileDeps` (or a sidecar JSON file at -`/.deps.json` — see §11 [TBD-2]) stores per-file: - -```json -{ - "src/main/java/com/example/Foo.java": { - "ext_hash": "sha256:abc...", // for change detection w/o git - "declares": ["com.example.Foo", "com.example.Foo.Bar"], - "injects": ["com.example.Service"], - "extends": ["com.example.Base"], - "calls": ["com.example.Service#run", ...], - "uses_anno": ["@RestController", "@CodebaseEndpoint"] - } -} -``` - -Inverse lookups are computed on demand or cached at write time — -implementation detail, not in this proposal's scope. - -### 2.5 Symmetric delete - -Every pass that *emits* nodes/edges keyed by source file must have a -symmetric `delete_*_for_file(path)` helper. Concretely: - -| Pass | Emits | Delete helper | -|------|-------|---------------| -| 1 | `Symbol`, `DECLARES` | `delete_symbols_for_file(p)` (cascades to DECLARES) | -| 2 | `EXTENDS`, `IMPLEMENTS`, `INJECTS` | per-edge filter on src-symbol's file | -| 3 | `CALLS` | per-edge filter | -| 4 | `Route`, `EXPOSES` | per-route filter on owner-symbol's file | -| 5 | `HTTP_CALLS`, `ASYNC_CALLS` (rows) | per-edge filter on src-symbol's file | -| 6 | (only mutates pass5 rows in place) | n/a — always reruns globally | - -Each helper takes a `Path` and a `kuzu.Connection`, executes the -DELETE Cypher, and returns a count for `--verbose` logging. None of -the existing tables need new columns — the file path is reachable -via the source `Symbol`'s `path` field. - -### 2.6 Change-detection sources - -Three sources, in priority order: - -1. **`--changed-paths` flag** (or stdin list). Caller responsibility - to be accurate. This is what the REFRESH proposal's decision engine - would pass. -2. **Git diff** between `--git-ref-base` and `HEAD`. Same logic as - REFRESH proposal §"Change Detection Strategy". -3. **Hash-based detection** using `FileDeps.ext_hash`. Walk the source - tree, compute hashes, compare against cached. Slowest of the three; - used as last-resort fallback. - -If all three fail or are ambiguous, fall back to full rebuild. - -## 3. Pass-by-pass incrementalisation notes - -### 3.1 Pass1 (parse) - -Trivial. Re-parse only files in the dirty set. No global state. - -### 3.2 Pass2 (cross-file edges) - -The edge `Foo EXTENDS Bar` lives in `pass2`. If only `Foo.java` -changed, the edge needs to be re-emitted with up-to-date metadata -(`Foo` may have changed which class it extends). If only `Bar.java` -changed, the edge target is unchanged but `Bar`'s symbol ID may have -new fields — re-emit. - -Concretely: pass2 re-runs over the dirty set, but only for *outgoing* -edges from symbols in dirty files. Inverse-EXTENDS lookup ensures -that when `Bar` changes, `Foo`'s edge is also refreshed — captured by -closure §2.3 rule 2. - -### 3.3 Pass3 (CALLS) - -Same shape as pass2. CALLS edges re-emitted for caller-symbols in -dirty files. Closure rule 3 catches inverse-CALLS. - -### 3.4 Pass4 (Routes) - -Two sub-passes: - -- **Extraction** is per-file. Easy. -- **Resolution** (PR-A2's three-strategy ladder) reads class-level - hints from a possibly-different file. Closure rule 6 covers this. - -Phantom-route cleanup currently runs in `pass6` (see PR-D3 review). -That stays as-is — phantoms are derived from the global graph state. - -### 3.5 Pass5 (HTTP_CALLS / ASYNC_CALLS caller-side) - -Per-file emit. Brownfield Layer 4 (`@CodebaseClient` / -`@CodebaseProducer`) overrides — closure rule 5 currently forces full -rebuild when an override file changes; that's pessimistic but safe. -A future PR-F or similar could implement file-scoped brownfield -closure to avoid this, once the Layer-4/5 fanout rules are explicitly -documented (see [TBD-1]). - -### 3.6 Pass6 (match outcomes + cross-service) - -**Always reruns globally.** Three reasons: - -1. Cross-service matching can flip outcomes service-wide when one - route's path template changes. Computing the closure of "which - pass5 rows might re-resolve" is O(|all routes|) anyway — might - as well rerun pass6. -2. Pass6 is fast: on `bank-chat-system` (15 files / 8 routes / 7 - call edges) it runs in ~1ms. Even at 10× scale we're still under - the human-perception threshold. -3. Pass6's idempotency contract (PR-E1 inline comment) was already - added with this proposal in mind. - -### 3.7 `_write_meta` - -Always rewritten — fields like `routes_total`, `cross_service_calls_total`, -`http_calls_match_breakdown`, `async_calls_match_breakdown` are aggregations -over the global graph state. Cost is negligible. - -## 4. CLI surface - -```bash -# Existing — unchanged -python build_ast_graph.py --source-root --kuzu-path [--verbose] - -# New — opt-in incremental -python build_ast_graph.py \ - --source-root --kuzu-path \ - --changed-paths \ - [--git-ref-base HEAD~1] \ - [--verbose] -``` - -Decision engine inside the script: - -1. If `--changed-paths` is given and `` exists with a - readable `FileDeps` index → attempt incremental. -2. If `` is empty/missing → full rebuild (no other choice). -3. If decision_engine.requires_full(changed) → full rebuild with a - logged reason. -4. If incremental fails mid-flight → roll back transaction, log, - fall back to full rebuild. - -## 5. MCP integration - -`refresh_code_index` (already in scope of the REFRESH proposal) gains -the ability to call the new incremental Kuzu path: - -```python -# Pseudocode inside server.py:refresh_code_index -decision = decide_mode(changed_paths, git_ref_base) -if decision.lance_mode == "incremental": - cocoindex_update(incremental=True, paths=decision.lance_paths) -else: - cocoindex_update(full=True) - -if decision.kuzu_mode == "incremental": - subprocess.run([..., "--changed-paths", decision.kuzu_paths_file]) -else: - subprocess.run([...]) # full rebuild -``` - -Decision engine returns two independent mode choices — LanceDB and -Kuzu may incrementally update independently. - -## 6. Determinism + correctness - -### 6.1 Determinism test extension - -Extend `tests/test_determinism.py` (or add `tests/test_incremental_equivalence.py`) -to assert: - -```python -def test_incremental_matches_full(tmp_path): - src = tmp_path / "src"; src.mkdir() - copy_fixture(src, "tests/fixtures/cross_service_smoke") - - # Full rebuild on initial state - run_full(src, kuzu_a := tmp_path / "a") - - # Mutate one file - (src / "svc-a/.../ClientA.java").write_text(modified_source) - - # Two paths to the same final state - run_full(src, kuzu_full := tmp_path / "full") - run_incremental(src, kuzu_inc := kuzu_a, changed=["svc-a/.../ClientA.java"]) - - assert all_sorted_ids(kuzu_full) == all_sorted_ids(kuzu_inc) - assert meta(kuzu_full) == meta(kuzu_inc) -``` - -This is the single most important test in the entire feature. -Without it, every incremental rebuild is a potential silent-divergence -risk. - -### 6.2 Failure modes - -- **Half-applied delete:** transaction wraps the entire incremental - pass; on any exception, ROLLBACK and fall back to full rebuild. -- **Corrupt FileDeps index:** detected by checksum; treated as - "incremental impossible, fall back to full". -- **Schema-version mismatch:** if `graph_meta.ontology_version` ≠ - current `ONTOLOGY_VERSION`, force full rebuild. - -## 7. Rollout - -1. **PR-T1 (foundation):** `FileDeps` table + `_write_dependency_index` - helper, written but unused. Determinism test extended (full - rebuild only, asserts FileDeps round-trip). -2. **PR-T2 (delete helpers):** symmetric `delete_*_for_file` for each - pass. Unit-tested in isolation. Still no incremental codepath. -3. **PR-T3 (incremental orchestrator):** `build_ast_graph_incremental` - function + `--changed-paths` flag. Pass-by-pass incremental - implementation. `test_incremental_matches_full` lit up here — must - pass for every fixture. -4. **PR-T4 (decision engine + CLI):** integrate the REFRESH proposal's - decision engine; wire `refresh_code_index` to dispatch. -5. **PR-T5 (brownfield closure refinement, optional):** narrow the - pessimistic "any brownfield-override change → full" rule once - Layer-4/5 fanout is explicitly documented. - -PR-T1 through PR-T4 are the headline. PR-T5 is an optimisation -on top. - -## 8. Risk assessment - -| Risk | Severity | Mitigation | -|------|----------|------------| -| Silent divergence between full and incremental | **High** | `test_incremental_matches_full` mandatory per fixture; covered by §6.1 | -| Pessimistic full-fallback hides real incremental wins | Medium | Track full-fallback rate via `graph_meta.last_rebuild_mode` + reason; review monthly | -| `FileDeps` index becomes stale | Medium | Stored in same DB transaction as graph; mismatch → full rebuild | -| Decision engine bugs cause unsafe incremental | High | Default-to-full on ambiguity; conservative initial rules; expand only after burn-in | -| Pass2/Pass3 closure misses an edge type | Medium | Determinism test surfaces this immediately | -| Performance regression (incremental slower than full for small repos) | Low | Heuristic: skip incremental when dirty set is >50% of files | - -## 9. Performance estimate - -On a hypothetical 1000-file Java repo: - -| Metric | Full rebuild | Incremental (1 file changed) | -|--------|--------------|-------------------------------| -| Pass1 parse | ~30s (1000 × 30ms) | <0.1s (1 × 30ms + 5-10 closure files) | -| Pass2-5 emit | ~20s | <0.2s | -| Pass6 match | ~1s | ~1s (always global) | -| `_write_meta` | <0.1s | <0.1s | -| **Total** | **~50s** | **~1.5s** | - -These are estimates — actual numbers depend on closure breadth and -Kuzu transaction overhead. Worth establishing a baseline benchmark -in PR-T1 (run full rebuild on `bank-chat-system` 10 times, record -mean/p95) and re-running after each PR. - -## 10. Open questions / explicit out-of-scope - -### [TBD-1] Brownfield closure granularity - -Layer-4 (`@CodebaseClient` / `@CodebaseProducer`) overrides today -operate via FQN matching. Editing the override file conceptually -invalidates every method that matches the override's FQN pattern. -The current proposal forces full rebuild in that case (closure rule -5). A more aggressive closure could parse the override and compute -exact dirty set. Defer to PR-T5 or a follow-up proposal once Layer-4 -fanout rules are formalised. - -### [TBD-2] FileDeps storage: in Kuzu vs sidecar JSON - -Two options: - -- **(a) New node table `FileDeps`** in the same Kuzu DB. Pros: - transactional with the graph, single artifact. Cons: Cypher - query overhead for inverse lookups; another schema column to - evolve. -- **(b) Sidecar `/.deps.json`** file. Pros: cheap - inverse-map computation in Python, no Cypher overhead. Cons: needs - manual atomicity (write-temp-rename pattern), separate version - field for migrations. - -Recommendation: **(b) sidecar JSON** — the file-level dependency -index is conceptually a build cache, not graph data. Decide in -PR-T1. - -### [TBD-3] Incremental for the LanceDB chunks? - -Already covered by CocoIndex's native incremental support -(`cocoindex update` without `--full-reprocess`). The REFRESH -proposal handles this. Out of scope here. - -### [TBD-4] Concurrent / interleaved rebuilds - -Single-writer assumed. If two `build_ast_graph.py` runs race against -the same `--kuzu-path`, behaviour is undefined. Document; do not -guard against. (Standard Kuzu single-writer constraint.) - -### [TBD-5] Watch-mode - -Out of scope. A follow-on proposal should layer a filesystem watcher -on top of this incremental path; it's not a free add-on because the -watcher needs debouncing, batched-change semantics, and its own state -machine. - -## 11. Done definition (proposal-level) - -This proposal is "ready for plan derivation" when: - -- [ ] All [TBD] items in §10 have either a decision or an explicit - "deferred to PR-T5+" tag. -- [ ] §3 closure rules have been validated against a real - multi-service fixture (`cross_service_smoke` is a candidate). -- [ ] §9 performance estimates have a measured baseline on at least - one fixture. -- [ ] Reviewer (you, principal engineer) approves §2.5 (symmetric - delete) as the right model — alternative is "wipe-and-rebuild - the dirty closure", which has similar correctness but worse - performance. -- [ ] Decision engine pseudocode in §4 + §5 is consistent with the - REFRESH proposal. - -When approved, derive `plans/PLAN-TIER2-INCREMENTAL-REBUILD.md` with -PR-T1 through PR-T5 broken out, and a per-PR Cursor task prompt set.