vector search

[zzqq131]

Hi, I saw that the commit wiped out the whole vector search and embedding module. I’m kinda curious—what’s the reasoning behind that? I’d love to hear your thoughts.

[andreinknv]

For anyone landing here looking for context — sharing what I dug up plus a complementary path.

Why it was removed (timeline)

• Mar 18 (fix: Lazy grammar loading to prevent V8 WASM OOM on large codebases #61) — Quantized embeddings shipped to fight V8 WASM Zone OOM crashes from Failed to initialize: Appears to be out of memory? #54.
• Apr 3 (feat: Always enable embeddings, remove config toggle #74) — Embeddings promoted to always-on, with concrete evidence that semantic search outperformed FTS for AI-assistant queries.
• Apr 7 — Issue indexing fails with 'Fatal process out of memory: Zone' #81 reports the same WASM Zone OOM persists on Node 25.x. Same day, the maintainer:
  • posted a diagnosis on indexing fails with 'Fatal process out of memory: Zone' #81 blaming V8 turboshaft + Node 25
  • committed 7507605 (Node 25+ runtime warning)
  • committed 453c39d removing all embeddings code

Stated reason: "Reduces package size and complexity while maintaining core static analysis capabilities." Implied (from the timing): @xenova/transformers ships ONNX as a second WASM toolchain on top of tree-sitter grammars, and V8's turboshaft Zone allocator can't handle JIT-compiling both. Cutting one of the two WASMs is the most reliable mitigation. (Apr 24 — cadam11 reports the same OOM on Node 24, confirming the underlying V8 issue isn't fully fixable from userspace alone.)

A complementary path: improve FTS without bringing back model deps

Most of what semantic search bought wasn't conceptual reasoning — it was robustness to:

• Partial-identifier queries: SQLite's default tokenizer treats getParser as one indivisible token, so a query for parser never matches it.
• Morphological variants: parsing vs parser vs parses.
• Natural-language noise: "how does X work" floods the BM25 ranking with stopword hits against any prose docstring.

Each of those is fixable in pure SQLite FTS5 without any model dependency. I prototyped + measured against codegraph's own src/, using the exact queries from #74:

Query	Baseline (current main)	After (subwords + Porter + stopwords)
ExtractionOrchestrator	#1	#1
how does file parsing work (target: getParser)	NOT FOUND in 20	#2
database connection management (target: DatabaseConnection)	#18	#1
resolves references between modules (target: resolveOne)	#19	#2

Mean rank: ~14 → 1.5.

I opened #104 with the full implementation: per-language subword splitting, Porter stemmer on the FTS table, stopword stripping reusing the existing STOP_WORDS set, plus a v3→v4 migration. 404/404 tests pass. Reviewed independently before pushing — three findings addressed (merged duplicate stopword sets, deduped subword tokens, made the migration idempotent on partial-DDL re-run).

This won't match true embeddings on purely conceptual queries with no shared identifier tokens (e.g., "code that talks to the database" finding nothing literally named database). But it recovers a large fraction of what was lost, with zero new dependencies, no install-time model download, no Node version constraints, no WASM allocator pressure. Up to the maintainer whether to take it; opening it as a no-pressure offer in the same direction the removal pointed.

[andreinknv]

#111 a bit different approach but works as well