Add semantic search layer on top of FTS5 to improve mem_search recall

[Basparin]

Context

Empirical finding from Basparin/Lawen project, 900-observation DB, session 2026-04-25.

mem_search is currently a wrapper over SQLite + FTS5 (lexical/tokenized search). Natural-language queries with valid intent but disjoint tokens silently return 0 results despite the content existing.

Reproducible evidence

Same DB, same content, ~5 minutes apart:

Query	Hits	Content existed?
workflow friction overuse optimization	0	Yes — observations on hooks/bridge/CLAUDE.md
tools rarely used skills inactive	0	Yes — skill registry observation
judgment-day ultrareview parallel review	0	Yes — referenced in session summaries
pulse statusline bridge	3	Same content as query #1, different tokens

Impact

The value of the saved corpus is gated by the caller's ability to guess past-self's word choices. For agents (Claude orchestrators) writing across many sessions and weeks, this leads to silent recall failures — saved decisions become invisible to queries with semantically equivalent but lexically distinct phrasing.

Proposal

Add an optional embeddings layer with hybrid scoring:

• FTS5 lexical match stays primary (deterministic, fast)
• Cosine similarity over embeddings augments — recovers synonymic / paraphrased queries
• Configurable weight (e.g. --semantic-weight 0.3) so users can tune lexical-vs-semantic mix
• Embedding model pluggable: local (sentence-transformers, ONNX) for offline, or API (Anthropic/OpenAI) for higher quality

Acceptance criteria

Same observation recoverable via at least 3 distinct natural-language queries that share intent but no overlapping tokens.

Backwards compatibility

Existing FTS5 behavior unchanged unless embeddings flag is enabled. Migration: re-embed existing observations on first opt-in run.

Workaround documented (no upstream change)

For now Basparin's symbiosis repo carries the workaround: mandatory topic_key per evolving topic, multi-keyword retry before concluding absent. Tracking: https://github.com/Basparin/symbiosis/pull/13

Related

Repo backend confirmed: Persistent memory system for AI coding agents. Agent-agnostic Go binary with SQLite + FTS5 (README).

[alexandervazquez98]

Confirmed — this affects me too. In a project with 481 observations, mem_search returned max 20 results that were never the most recent — only those FTS5 ranked highest by keyword density.

The core issue: FTS5 rank != recent OR relevant to the agent current context. A query like bug fix returns ancient results mentioning those words many times, leaving out newer memories that use different vocabulary.

Evidence: to get context I had to use sqlite3 direct and bypass mem_search entirely. That not viable for an autonomous agent.

+1 on the embeddings layer proposal — it would be transformative for real recall.

Related: Issue #146 (MCP server search returns empty) has the same root cause — FTS5 silently fails to retrieve content that exists.

[Basparin]

Confirmed — this affects me too. In a project with 481 observations, mem_search returned max 20 results that were never the most recent — only those FTS5 ranked highest by keyword density.

The core issue: FTS5 rank != recent OR relevant to the agent current context. A query like bug fix returns ancient results mentioning those words many times, leaving out newer memories that use different vocabulary.

Evidence: to get context I had to use sqlite3 direct and bypass mem_search entirely. That not viable for an autonomous agent.

+1 on the embeddings layer proposal — it would be transformative for real recall.

Related: Issue #146 (MCP server search returns empty) has the same root cause — FTS5 silently fails to retrieve content that exists.

My Claude got too frustrated with that, that it uploaded this issue autonomously lol

[alexandervazquez98]

Search Recall Enhancement — Proposed Implementation Plan

Based on analysis of the codebase and user experiences, here is a comprehensive roadmap to improve mem_search recall:

---

Problem Summary

1. FTS5 rank = bm25 — prioritizes keyword density, not relevance to current context
2. Hard cap at 20 results — with 481+ observations, recent memories are pushed out
3. No recency weighting — new observations with different vocabulary are invisible
4. Cloud dashboard has no FTS — only strings.Contains on title+type

---

Solutions

Option A: FTS5 Ranking Improvements

What: Optimize existing FTS5 without adding embeddings

Improvement	Description
BM25F field boosting	Title matches 3x, tool_name 2x over content
Recency boost	Decay factor favoring recent observations
Query expansion	Synonym mapping (fix to bug, auth to login)
MMR diversification	Avoid returning too-similar results

Impact: +20-30% recall improvement
Risk: Low — purely additive, no breaking changes
Cloud impact: None — cloud does not use FTS5

---

Option B: Hybrid Search with Embeddings

What: Add semantic similarity on top of FTS5 via embeddings

Query: "password handling flow"
         |
    +----+----+
    |         |
    v         v
 +------+ +-------+
 | FTS5 | |Embed- |
 | bm25 | |  ding |
 +--+---+ +---+---+
    |        |
    +----+---+
         v
  +-----------+
  | Reciprocal|
  | Rank Fusion|
  |  (k=60)   |
  +-----------+

New MCP parameter:

mem_search(query: "auth flow", semantic_weight: 0.3)
// 0.0 = pure FTS5 (backward compatible)
// 0.3 = 30% semantic, 70% lexical

Embedding model: all-MiniLM-L6-v2 (384 dims, ~90MB, ~50ms/obs)
Storage: ~1.5KB per observation. 10K obs = ~15MB

Impact: +60-80% recall improvement
Risk: Medium — ONNX integration, write latency +50-100ms
Cloud impact: Decision required

• Local-only: cloud users get FTS only
• Sync embeddings: +1.5KB x N per sync cycle
• Compute on-demand: requires cloud ML infra

---

Option C: mem_recent Tool

What: New tool returning recent observations by date (no FTS)

mem_recent(project: "my-project", limit: 50)

Impact: Immediate visibility into recent memories
Risk: None — purely additive tool
Cloud impact: None

---

Recommended Roadmap

Phase 1: mem_recent (Option C)
  - Quick win — solves immediate "where are my recent memories" problem
  - Very simple implementation

Phase 2: FTS5 Ranking Improvements (Option A)
  - BM25F with field boosting
  - Recency boost
  - Query expansion with synonyms
  - No new dependencies, low risk

Phase 3: Hybrid Search (Option B)
  - If Phase 2 proves insufficient
  - Embedding computation pipeline
  - RRF fusion algorithm
  - Start local-only (cloud sync deferred)

---

Key Decisions Needed

Question	Options
Embeddings local-only or synced?	Local-only (simpler) / Sync (complex, costly)
Accept ~50ms write latency per new obs?	Yes (better search) / No (keep it fast)
Cloud users need semantic search?	Yes (requires infra) / No (local-only is fine)
Embedding model size?	Smaller=faster / Larger=more accurate

---

What Would Break

Change	Breaking Impact
BM25F ranking	Result order may change slightly
semantic_weight param	None — backward compatible, default 0.0
New embedding table	None — additive schema
Background embedding computation	+50ms latency on new observations

---

Recommendation: Start with Phase 1 (mem_recent) + Phase 2 (FTS5 improvements) as immediate wins. Pursue Phase 3 (hybrid) only if FTS5 optimizations prove insufficient.

Happy to help implement any of these phases.

[Basparin]

Update — additional empirical evidence + concrete Phase 1 spec (2026-04-29)

Cross-session continuation from the same Basparin corpus. 1131 observations now (+231 since original report). Problem reproduces and is consistent.

Section A — Additional A/B testing (same DB, same 5-min window)

Query (FAIL)	Tokens	Hits	Same intent (PASS)	Tokens	Hits
session start protocol initialization instructions engram	8	0	loop nocturno (different topic, present in corpus)	2	3
untracked perdidos pull restart cortex.js daemon working tree	8	0	untracked files volatile	3	1
claude web review architecture astronaut bias 70 30 codigo diseño	10	0	arch astronaut	2	0 (vocab not in saves — different problem)
PR 98 user reply fix sub-agent cortex	7	0	PR 98	2	3 (cross-project pollution — see below)
mem_session_start mem_context boot session arranque	6	0	topic_key direct: thronglets/lesson-untracked-files-volatile-2026-04-29	—	1

Pattern reproduced: queries with 7+ mixed-language tokens consistently 0-hit even when the underlying observation exists. Confirms BM25 with no query expansion penalizes long token vectors against short documents.

Secondary finding: PR 98 matches PR #98 from a different project (gentle-ai) with no implicit project: filter — cross-project pollution affects multi-project users. Related to #171 / #172.

Section B — Concrete Phase 1 spec — mem_recent

Aligned with @alexandervazquez98's roadmap. Proposing the MCP shape for review:

Tool name: mem_recent

Signature:

mem_recent({
  project?: string,         // optional filter; default = all
  scope?: 'project' | 'personal',
  limit?: number,           // default 20, max 50
  type?: ObservationType,   // optional filter
  since?: string            // ISO 8601, optional — observations newer than this
})

Behavior:

• Pure recency-ordered query (ORDER BY created_at DESC)
• No FTS5 invocation — bypasses BM25 ranking entirely
• Same result shape as mem_search (compatible with existing mem_get_observation flow)

Use case: agent at session start asks "what happened recently on this project". Currently impossible — every FTS query is filtered by token relevance, so recent observations using new vocabulary become invisible.

Implementation surface (best estimate from external view):

• New handler: internal/mcp/handlers/recent.go (or wherever search lives)
• Storage query: simple SELECT ... ORDER BY created_at DESC LIMIT N against observations
• MCP tool registration: 1 entry
• Tests: unit (recency order, filters) + integration (end-to-end MCP roundtrip)

Estimated effort: effort:small — fits within < 1h. Good first-contribution candidate.

Cross-link

Complemented by #137 (mandate mem_context at session start). Even with semantic search added, an agent that doesn't call the search at session-start still operates blind. Both ship together for full effect.

Awaiting status:approved

Per CONTRIBUTING — not opening PR until issue has status:approved. Happy to implement Phase 1 (mem_recent) immediately upon approval.