test(rag): rerank acceptance tail — ACL regression test + latency budget (#215)#242
Merged
Conversation
…sion test + latency budget The cross-encoder rerank implementation merged earlier (#221/#226/#229/#236); this closes the verification tail of #215: - The issue's missing regression test, at the server level (crates/escurel-server/tests/fusion_acl.rs:: reranked_search_never_surfaces_rows_hidden_pre_rerank): with the rerank stage ACTIVE on the real gateway search path, a reranked result set never contains a row the caller could not see pre-rerank — even when the forbidden rows (cross-owner + unresolved-owner sql_view deals) are the reranker's highest-scoring candidates. A rerank-off control process over identical data pins set equality (the reranked set is exactly the ACL-filtered set, reordered), and a promoted entitled page at rank 1 proves the stage really ran. Uses a deterministic KeywordReranker via Indexer::with_reranker (mirrors crates/escurel-index/tests/rerank.rs), so it runs under default features — no model download. - docs/eval/rerank-latency-budget.md: the latency budget the issue asked for, from the two committed eval runs (baseline-scifact.md + raw JSON) — rerank_candidates (default 100) bounds the worst case; measured ~15 s/query (100 snippet candidates) and ~63 s/query (50 full-body candidates) on CPU vs ~130-150 ms first-stage p50. Linked from docs/eval/README.md. Closes the remaining acceptance boxes of #215. The issue's third config mode `ce_head` was never implemented anywhere (no code, no history); the roadmap defers it and its precondition (EmbeddingGemma in candle) does not exist yet — tracked in the issue discussion, not in code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Finishes the verification tail of #215 (the reranker implementation itself is already on main):
reranked_search_never_surfaces_rows_hidden_pre_rerank(escurel-server/tests/fusion_acl.rs): through the real server search path, the forbidden rows are deliberately the reranker's maximum-relevance candidates — any 'rerank before ACL' or 'rerank re-injects candidates' regression would surface them at rank 1. Also pins set-preservation vs a rerank-off control (reordered, never widened) and that the stage actually ran (a mid-pack entitled page promoted to rank 1). DeterministicKeywordRerankervia theConfigOverrides::indexerseam — default features, no model download, runs everywhere.docs/eval/rerank-latency-budget.md(linked from docs/eval/README.md) —rerank_candidatesbounds worst case; measured figures from the existing baselines (fix(eval): apply rerank in the query path + SciFact baseline #235/docs(eval): record full-passage rerank re-measure (#236 confirmed) #237): ~1.3 s/pair full-body CPU → p50 ~63 s/query at 50 candidates, vs ~130–150 ms first-stage; guidance: CPU deployments rerank off or 10–20 candidates; the real fix is GPU/BLAS — and RAG: Matryoshka two-pass retrieval — coarse HNSW shortlist at low dim, full dim + rerank on survivors #218's two-pass is the designed buy-back.Test plan
Red observed first (reranker detached → rank-1 assertion fails for the right reason), then green. Full local gate: fmt --check ✓, clippy -D warnings ✓, test --workspace --all-targets ✓ (30/30 binaries), release build ✓; plus the feature-gated cross-encoder suite (3/3).
🤖 Generated with Claude Code
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.