perf(search): parallelize cross-source searches and cache query embeddings#119
Merged
jeffgreendesign merged 7 commits intoJul 11, 2026
Merged
Conversation
…dings Two low-risk runtime wins on the search hot path, plus a CI cleanup: - unifiedSearch ran document -> memory -> conversation searches serially even though all three depend only on the query embedding. Fuse them into one Promise.all so their DB round-trips overlap. Output shape and scoring are unchanged. - Add a bounded (LRU + TTL) query-embedding cache so repeated queries skip the external embedding provider round-trip. Keyed by provider+model so a config change can't return a wrong-dimension vector; only successful embeddings are cached. Covers all three sources since they share the query embedding. - Add concurrency groups with cancel-in-progress to the CI, CodeQL, and docs-deploy workflows so superseded pushes stop wasting runners. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PG8KhXDuHgAqNEcvLVXxi6
…tor 0.8 iterative scans Fixes a correctness bug: filtering tags/sourceType/contentType in JS after fetching only the top limit*3 RRF chunks could return 0 results even when matching documents exist, because the matching chunks never entered the fetch window. - hybrid_search() now takes filter_source_type / filter_content_type / filter_tags and applies them inside BOTH the full_text and semantic CTEs (before their limit), so selective filters are never starved. Replicated across all four dimension variants (1536/3072/1024/768), which stay identical modulo vector(N). Old 6-arg signature is dropped first to avoid an ambiguous overload. - Add a GIN index on documents.metadata (jsonb_path_ops) to back the tag / content_type predicates. - Enable pgvector 0.8 hnsw.iterative_scan per connection (exception-guarded so older pgvector still connects), so the filtered HNSW scan keeps going until the LIMIT is met. Also benefits memory search's existing entity_types filter. - unifiedSearch forwards the categorical filters to SQL and drops the JS post-filters for them; minScore stays in JS (it also gates weighted cross-source scores) and only it now widens the fetch. - Add a dated idempotent migration and note pgvector >= 0.8 in the install docs. - New tests for the DB param mapping and the service-level filter pushdown. Validated the SQL end-to-end against a real Postgres 16 (pgvector bits stubbed): filter predicates, tag AND-semantics, the 9-arg signature, and the GIN index all behave as intended. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PG8KhXDuHgAqNEcvLVXxi6
…search Finishes the two remaining Tier-2 runtime items. #5 HNSW ef_search: - Set hnsw.ef_search per connection (default 100, env HNSW_EF_SEARCH), alongside the existing iterative_scan GUC, in the same exception-guarded DO block so older pgvector still connects. Read via a local envInt() helper to keep the DB layer decoupled from config. Raises recall over pgvector's built-in default of 40. Validated against a real Postgres 16: the DO block swallows the unknown-GUC error on plain PG and leaves the connection usable. #6 ingest throughput: - Embedding batches within one generateEmbeddings call now run with bounded, order-preserving concurrency via a shared embedBatchesConcurrently() helper (pLimit + Promise.all + flat). Applied to all three providers; OpenAI keeps its index re-sort. Biggest win for semantic chunking and large single embeds. - embedAndStoreChunks runs its 128-chunk embed+insert windows with bounded concurrency, overlapping embed and insert for large documents. Each chunk keeps its own chunkIndex so insert order is irrelevant; peak memory stays bounded to ~concurrency windows. - Two new int knobs (EMBEDDING_BATCH_CONCURRENCY, EMBED_WINDOW_CONCURRENCY, both default 2); documented in .env.example and docs/ENV.md. Tests: unit-test the concurrency helper (order preserved under reverse-completion, respects the bound) and embed-store (correct chunkIndex<->embedding pairing across concurrent windows, length-mismatch guard still throws). Updated the upload-processor config mock for the new keys. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PG8KhXDuHgAqNEcvLVXxi6
Bump root zod ^3.23.0 -> ^4.4.3. The lockfile now dedups to a single zod@4.4.3 instance shared by @modelcontextprotocol/sdk (1.29, declares zod ^3.25 || ^4.0), @modelcontextprotocol/ext-apps, @anthropic-ai/sdk, and zod-to-json-schema — so the tool schemas (passed as raw zod objects to registerTool) and the SDK share one instance, avoiding the dual-instance hazard. The openai@4.104 peer warning for zod ^3 is inert; we only use OpenAI for embeddings, not its zod-based helpers. Modernize the only two deprecated-in-v4 spots: - z.string().url() -> z.url() (5 sites in config + url tool) - result.error.format() -> z.treeifyError() in config validation logging Every other zod usage was already v4-correct (.issues accessors, two-arg z.record, positional custom messages, .default().transform() chains). Verified: typecheck, 444 tests, esbuild build, createMcpServer tool registration, and a live MCP tools/list over HTTP (all 25 tools serialize their schemas to JSON Schema with no error). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PG8KhXDuHgAqNEcvLVXxi6
pdf-parse 1.1.x is long-unmaintained and runs a debug self-test on import.
Swap it for unpdf (maintained, pdfjs-based) on the server + CLI extraction
paths. API surface is tiny — buffer in, text out:
- src/services/processor/handlers/pdf.ts and the takeout CLI converter now call
unpdf's extractText(new Uint8Array(buffer), { mergePages: true }).
- Drop pdf-parse from the root package.json (add unpdf); delete the obsolete
ambient src/types/pdf-parse.d.ts (unpdf ships its own types).
- Update the registry test mock to unpdf.
The desktop app intentionally keeps pdf-parse for now — its Electron main
bundle (esbuild externals + electron-builder packaging) can't be verified in
this environment.
Verified: typecheck, 444 tests (incl. the registry PDF path), esbuild build,
and a real end-to-end extraction (unpdf pulled "Hello Textrawl PDF" out of a
generated one-page PDF).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PG8KhXDuHgAqNEcvLVXxi6
Bump @biomejs/biome 1.9.4 -> ^2.0.0 (resolves 2.5.3) in root, website, and dashboard, kept in lockstep. Ran `biome migrate` to convert biome.json to the v2 shape: $schema bump, organizeImports -> assist.actions.source.organizeImports, files.ignore -> files.includes (negated globs), overrides[].include -> includes. Biome 2's expanded recommended set surfaced new findings; reconciled to keep CI green while preserving the project's existing lenient posture: - Real fixes: two forEach callbacks that implicitly returned a value (useIterableCallbackReturn) in the html + takeout converters and the desktop file-router now use block bodies. - Excluded non-source assets/fixtures from linting (public/**, __tests__/fixtures). - Softened brand-new v2 rules the project never enforced to warn/off, matching how it already softens noExplicitAny/noForEach/noNonNullAssertion: noArrayIndexKey, useButtonType, noStaticElementInteractions, noAmbiguousAnchorText, noSvgWithoutTitle (warn); noImportantStyles (off — !important is intentional in the docs/dashboard CSS). - Applied v2's safe autofixes: dead-import/var removal, parseInt radix, optional chaining, and the new import-sort order across the repo (mechanical, no semantic change — keeps the tree consistent with `biome check --write`). Remaining 24 warnings are minor pre-existing suggestions (unused params, array index keys) newly surfaced by v2; non-blocking (biome check exits 0). Verified: typecheck, 444 tests, esbuild build, and `biome check` exit 0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PG8KhXDuHgAqNEcvLVXxi6
Contributor
|
Important Review skippedNo new commits to review since the last review. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughThis change adds filtered hybrid-search support with metadata indexes and pgvector tuning, bounded concurrency for embedding and chunk storage, and cached query embeddings. PDF extraction moves from Sequence Diagram(s)sequenceDiagram
participant Client
participant UnifiedSearch
participant QueryEmbeddingCache
participant HybridSearch
participant PostgreSQL
Client->>UnifiedSearch: Submit search query and filters
UnifiedSearch->>QueryEmbeddingCache: Get query embedding
QueryEmbeddingCache-->>UnifiedSearch: Return cached or generated embedding
UnifiedSearch->>HybridSearch: Pass embedding and metadata filters
HybridSearch->>PostgreSQL: Execute filtered hybrid_search query
PostgreSQL-->>HybridSearch: Return ranked results
HybridSearch-->>UnifiedSearch: Return document results
UnifiedSearch-->>Client: Return unified search response
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/cli/converters/takeout.ts`:
- Around line 30-34: Update the unhandledRejection handler to route its warning
through the existing logger by replacing the direct console.error call with
logger.error, while preserving the current message and continue-on-rejection
behavior.
In `@scripts/migrations/2026-07-10-hybrid-search-filters.sql`:
- Line 22: Update the documents_metadata_gin_idx creation statement to use
CREATE INDEX CONCURRENTLY, and ensure this migration runs outside a transaction
because PostgreSQL does not allow concurrent index creation within one.
In `@scripts/setup-db.sql`:
- Around line 90-92: Update the JSONB filter predicates in both CTEs to use
top-level metadata containment with d.metadata @> jsonb_build_object(...),
replacing the content_type extraction equality and nested tags containment
checks while preserving their existing null-filter behavior.
In `@src/services/__tests__/embed-store.test.ts`:
- Around line 49-55: Add an assertion to the embedding-count mismatch test for
embedAndStoreChunks that verifies createChunks was not called after the
rejection. Keep the existing rejection assertion and ensure the mock is
available for checking this promised no-write behavior.
In `@src/services/embed-store.ts`:
- Around line 35-59: The chunk-writing flow around generateEmbeddings,
createChunks, and Promise.all must prevent partial persistence when any window
fails. Add cancellation or compensating cleanup for already-written windows,
ensuring no createChunks operation remains able to commit after the first
failure and the overall upload leaves no partial chunk rows.
In `@src/utils/config.ts`:
- Around line 52-63: Update EMBEDDING_BATCH_CONCURRENCY and
EMBED_WINDOW_CONCURRENCY to use z.coerce.number().int().min(1) instead of string
parsing with parseInt, ensuring malformed and fractional environment values are
rejected. Preserve their existing default of 2 and do not add a maximum
constraint unless an established concurrency ceiling exists.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: a0c3d01b-0140-49f9-8c9e-4e2683553554
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (93)
.env.example.github/workflows/ci.yml.github/workflows/codeql.yml.github/workflows/deploy-website.ymlbiome.jsondashboard/app/insights/page.tsxdashboard/app/knowledge/page.tsxdashboard/app/memory/page.tsxdashboard/app/page.tsxdashboard/app/upload/page.tsxdashboard/lib/__tests__/api.test.tsdashboard/lib/api.tsdashboard/lib/queries.tsdashboard/package.jsondesktop/src/main/index.tsdesktop/src/main/services/document-processor.tsdesktop/src/main/services/file-router.tsdesktop/src/main/services/project-manager.tsdesktop/src/main/utils/run-cli-script.tsdocs/ENV.mddocs/getting-started/installation.mdxpackage.jsonscripts/cli/converters/eml.tsscripts/cli/converters/facebook.tsscripts/cli/converters/html.tsscripts/cli/converters/instagram.tsscripts/cli/converters/mbox.tsscripts/cli/converters/reddit.tsscripts/cli/converters/spotify.tsscripts/cli/converters/takeout.tsscripts/cli/lib/analyze.tsscripts/cli/lib/normalizer.tsscripts/cli/lib/security.tsscripts/cli/lib/splitter.tsscripts/cli/scan.tsscripts/cli/split.tsscripts/cli/upload.tsscripts/migrations/2026-07-10-hybrid-search-filters.sqlscripts/setup-db-google.sqlscripts/setup-db-ollama-v2.sqlscripts/setup-db-ollama.sqlscripts/setup-db.sqlscripts/ui/server.tssrc/api/__tests__/upload-process.test.tssrc/api/insight-routes.tssrc/api/middleware/rateLimit.tssrc/api/oauth/jwt.tssrc/api/oauth/routes.tssrc/api/routes.tssrc/api/upload-process.tssrc/api/upload-sessions.tssrc/db/__tests__/search.test.tssrc/db/memory-relations.tssrc/db/memory-search.tssrc/db/pg-client.tssrc/db/search.tssrc/db/stats.tssrc/index.tssrc/server.tssrc/services/__tests__/embed-store.test.tssrc/services/__tests__/embeddings.concurrency.test.tssrc/services/__tests__/search.test.tssrc/services/__tests__/upload-processor.test.tssrc/services/embed-store.tssrc/services/embeddings.tssrc/services/memory-extraction.tssrc/services/pg-analyze/formatter.tssrc/services/pg-analyze/history.tssrc/services/processor/__tests__/archive-zip.test.tssrc/services/processor/__tests__/registry.test.tssrc/services/processor/handlers/pdf.tssrc/services/search.tssrc/services/upload-processor.tssrc/tools/index.tssrc/tools/insights.tssrc/tools/memory.tssrc/tools/pg-analyze.tssrc/tools/url.tssrc/types/index.tssrc/types/pdf-parse.d.tssrc/ui/index.tssrc/utils/__tests__/query-embedding-cache.test.tssrc/utils/__tests__/source-span.test.tssrc/utils/config.tssrc/utils/query-embedding-cache.tswebsite/app/(home)/layout.tsxwebsite/app/(home)/page.tsxwebsite/app/api/search/route.tswebsite/app/docs/[[...slug]]/page.tsxwebsite/app/docs/layout.tsxwebsite/app/layout.config.tsxwebsite/components/landing/hero.tsxwebsite/package.json
💤 Files with no reviewable changes (3)
- src/types/pdf-parse.d.ts
- src/db/stats.ts
- src/index.ts
- hybrid_search JSONB filters now use top-level containment so the new documents_metadata_gin_idx is actually used: metadata @> jsonb_build_object( 'content_type'|'tags', ...) instead of metadata->>'content_type' = ... and metadata->'tags' @> ... (the latter can't use a GIN index on `metadata`). Validated against Postgres 16: identical result sets, and the new form plans a Bitmap Index Scan (incl. via a parameterized statement) where the old form seq-scans. Applied to all four setup-db*.sql variants + the migration. - Migration builds the GIN index with CREATE INDEX CONCURRENTLY (documented as autocommit-only) so it doesn't block writes on a live documents table. - mbox converter: drop the dead _prevLineEmpty assignments (CodeQL useless-assignment). - takeout converter: route the unhandledRejection warning through logger.error instead of console.error, per the logging guideline. - embed-store test: assert createChunks is not called on an embedding-count mismatch (locks in the no-partial-write guard). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PG8KhXDuHgAqNEcvLVXxi6
Owner
Author
|
Addressed the review in
@coderabbitai review Generated by Claude Code |
Contributor
|
✅ Action performedReview finished.
|
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two low-risk runtime wins on the search hot path, plus a CI cleanup:
though all three depend only on the query embedding. Fuse them into one
Promise.all so their DB round-trips overlap. Output shape and scoring are
unchanged.
external embedding provider round-trip. Keyed by provider+model so a config
change can't return a wrong-dimension vector; only successful embeddings are
cached. Covers all three sources since they share the query embedding.
docs-deploy workflows so superseded pushes stop wasting runners.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01PG8KhXDuHgAqNEcvLVXxi6