Flagship correctness fixes, real escalation metric, lean install

[ChunkyTortoise]

Summary

A focused, conflict-free, code-only change (single commit rebased onto current main):

• reranker.py — stop mutating the caller's SearchResult objects in place (this corrupted the original retrieval scores still used upstream in agentic_rag); return model_copy'd results so rerank is pure w.r.t. its inputs.
• model_ab_test.py — the standard error was the Welch (unpooled) SE of the mean difference but was named pooled_se. Renamed se_diff; math unchanged.
• agentic_rag.py — dedup used chunk_id or doc_id, which collapses distinct chunks when chunk_id is a falsy-but-valid value (""). Now chunk_id is not None, extracted to a _dedup_key helper.
• metrics.py — escalation_rate was a hardcoded 0.12 (also surfaced in the Quality Monitor dashboard). Now computed from ExtractedRecord.needs_review via a single count(*) FILTER (WHERE ...) query.
• Lean install — extract-msg (Outlook .msg) pulls red-black-tree-mod, which needs a C toolchain and broke pip install on clean environments. Made opt-in via requirements-msg.txt; the code already degrades gracefully (HAS_EXTRACT_MSG).

Scope note

This branch originally also carried a doc-honesty pass and a self-verifying repo_stats.py/CI mechanism. While it was in progress, main independently shipped a more extensive parallel pass (hiring narrative, metric-provenance labelling, offline-replay eval gate). That work is now redundant, so it was dropped rather than force a conflicted merge — leaving only the uniquely-valuable, non-conflicting code fixes. Verified that none of main's recent work is reverted.

Validation

• 1,104 unit tests pass on top of current main (includes main's own new tests), 0 failures, ruff clean.
• The new count(*) FILTER aggregate compiles to valid PostgreSQL.
• Clean-venv pip install -r requirements_ci.txt exits 0.
• Integration suite (Postgres/Redis) not run in the authoring env — covered in CI.

Test plan

• CI green: full pytest incl. integration (test_business_metrics.py now asserts the computed rate), ruff, eval-gate
• Sanity-check /metrics/business and /metrics/quality-trend return a real computed escalation_rate
• Confirm .eml extraction still works with extract-msg absent; .msg returns the graceful error

https://claude.ai/code/session_016hjhwGkKkA7duB6cjzfQ2y

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fcc4ee4a05

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve doc identity for empty chunk IDs

When a tool returns chunk_id="" (which the new comment explicitly treats as a valid falsy chunk id), this key becomes the same empty string for every result, so _merge_results/_deduplicate will drop all but the first such result even when they come from different doc_ids. Use a composite key that still includes doc_id when chunk_id is present but empty, or normalize empty strings to absent if they are not actually valid.

Useful? React with 👍 / 👎.