Follow-up from #87 (hybrid search, Phase 1).
Problem
semantic_search_sessions (sqlite-vec KNN) has no relevance threshold — it always returns the k nearest vectors regardless of how far they are. So a hybrid query that matches no keyword still surfaces sessions purely by nearest-neighbour distance, even when nothing is genuinely relevant. Most visible on small archives / nonsense queries; on a large archive the RRF fusion dilutes it, but the noise is real.
Fix
Add a max-distance cutoff to semantic_search_sessions: drop vec hits whose distance exceeds a threshold before fusion. Needs calibration against the bge-small distance distribution on the real archive (pick a percentile, not a magic constant). Optionally expose it as LORE_SEMANTIC_MAX_DISTANCE.
Also worth considering (deferred from #87 Phase 1)
- Message/chunk-level embeddings — fills the reserved
entity_type='message' FTS rows; far more precise than per-session vectors but a much larger index. See docs/PLAN-hybrid-search.md non-goals.
- bge-small truncates ~512 tokens, so per-session vectors only capture the head of long sessions — chunk-level would fix that too.
Ref: lore/storage/search.py::semantic_search_sessions, docs/PLAN-hybrid-search.md.
Follow-up from #87 (hybrid search, Phase 1).
Problem
semantic_search_sessions(sqlite-vec KNN) has no relevance threshold — it always returns the k nearest vectors regardless of how far they are. So a hybrid query that matches no keyword still surfaces sessions purely by nearest-neighbour distance, even when nothing is genuinely relevant. Most visible on small archives / nonsense queries; on a large archive the RRF fusion dilutes it, but the noise is real.Fix
Add a max-distance cutoff to
semantic_search_sessions: drop vec hits whosedistanceexceeds a threshold before fusion. Needs calibration against the bge-small distance distribution on the real archive (pick a percentile, not a magic constant). Optionally expose it asLORE_SEMANTIC_MAX_DISTANCE.Also worth considering (deferred from #87 Phase 1)
entity_type='message'FTS rows; far more precise than per-session vectors but a much larger index. Seedocs/PLAN-hybrid-search.mdnon-goals.Ref:
lore/storage/search.py::semantic_search_sessions,docs/PLAN-hybrid-search.md.