Skip to content

perf(graph): skip full global pass-1 re-parse in increment when no Clients/Producers exist #379

Description

@HumanBean17

Problem

incremental_rebuild runs a full pass1_parse of every source file to rebuild the global tables for pass 5/6 (build_ast_graph.py, the "Step 6: Global pass 5-6" block — pass1_parse(source_root, tables_for_global, ...) followed by global pass 4/5/6), even when only one file changed. This was added for correctness in #352 / f902fb7: pass 5 links Feign HTTP_CALLS to routes and pass 6 matches cross-service edges, both of which need project-wide context.

For repositories with no imperative HTTP clients (no Feign / RestTemplate / WebClient / Kafka client-or-producer), pass 5/6 emit zero edges, so the full re-parse (and the global pass 4/5 that follows it) is pure waste on every increment.

Evidence (Shopizer, 1167 Java files)

[graph] pass 5 · HTTP_CALLS: 0 edges, ASYNC_CALLS: 0 edges
[graph] pass 6 · http_match={}, async_match={}, cross_service_calls_total=0

Timed on a cold graph run:

  • pass 1 full re-parse: ~5.3s
  • global pass 4 + pass 5: ~3s

~8s spent on every increment producing no output, on a repo class that has no imperative HTTP/async clients.

Proposal

Guard the global pass-5/6 block: only run it (and the full pass1_parse that feeds it) when the codebase actually contains Client / Producer nodes. Cheapest detection — query the existing graph before deciding:

has_clients  = conn.execute("MATCH (c:Client) RETURN count(*)").get_next()[0] > 0
has_producers = conn.execute("MATCH (p:Producer) RETURN count(*)").get_next()[0] > 0
if not (has_clients or has_producers):
    # skip global pass1 + pass4 + pass5 + pass6 + _write_clients_producers_and_calls

The scoped pass 1-4 already ran on the changed files + dependents, so the graph stays correct for everything except cross-service HTTP/async edges — which don't exist when there are no clients/producers. This preserves the #352 correctness fix for repos that DO have clients while skipping the work for the common case.

Fallback if a DB count is undesirable: a fast annotation scan for client/producer indicators during the scoped pass.

Related (lower priority, separate)

Graph pass 1 (node registration, ~5.3s) and pass 3 (call resolution, ~8.3s) are single-threaded Python loops over files. parse_java itself is cheap (~0.9s total — Rust tree-sitter, per-thread Parser); the cost is the Python node/call work around it, which mutates shared GraphTables. Parallelizing via threads (per-file accumulation + merge into tables) would cut the ~14s combined further. Tracked separately; this issue is specifically the increment guard.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions