vouch can measure retrieval quality — eval/recall.py scores recall over a labeled query set (#226), and the embeddings evals check semantic fidelity. what it cannot answer is the question an operator actually cares about: is this kb helping, and which specific claims earn their keep? a claim can be perfectly retrievable and still be dead weight, or actively misleading. today nothing correlates "artifact X was surfaced into a session" with what happened in that session, so a reviewer deciding what to expire is flying blind.
this proposes a read-only, measurement-only effectiveness signal: per approved artifact, the association between it being in a session's context pack and a coarse session outcome, reported with a confidence interval and a deliberately conservative verdict.
proposed surface
vouch health effectiveness [--window 90d] [--min-samples N] [--format text|json]
# same surface as kb.effectiveness over mcp/jsonl
- reads two things already on disk: which artifacts were surfaced (context-pack composition, recordable from the read path into
state.db), and a per-session outcome signal derived from the append-only audit log (audit.read_events) — e.g. a session that ended with confirm/cite activity vs. one that ended in contradict/reject.
- computes, per artifact: surfaced-with-good-outcome vs. surfaced-with-bad-outcome, an associational lift, and a 95% Wilson confidence interval.
- verdict is gated by statistical power:
useful / harmful only when the interval clears baseline and the sample meets --min-samples; otherwise unverified / insufficient. an untrustworthy number never renders as a confident verdict.
- output ranks artifacts by earned value so a reviewer can see which approved claims to promote, re-cite, or
expire.
a src/vouch/eval/effectiveness.py module composes metrics.compute + audit.read_events + the context-pack surfacing log. the surfacing log is a new derived table in index_db.SCHEMA (a cache, rebuildable — the yaml stays source of truth), added to index_db.reset.
review gate & scope
nothing here writes or edits knowledge and it never mutates trust or lifecycle — it reads the audit log and a derived cache and reports. it explicitly does not auto-expire or auto-approve on a bad verdict; the human at the gate decides. the outcome signal is derived from events that already exist (the audit stream is authoritative), so there's no new authoritative state. if exposed as kb.effectiveness, it touches the four registration sites (server.py, jsonl_server.py, capabilities.py, cli.py) with a test at tests/test_effectiveness.py.
acceptance criteria
related: #226 (recall-quality eval — this is the outcome layer above it), #318 (synthesis faithfulness eval), #101 / #164 (metrics / stats).
vouch can measure retrieval quality —
eval/recall.pyscores recall over a labeled query set (#226), and the embeddings evals check semantic fidelity. what it cannot answer is the question an operator actually cares about: is this kb helping, and which specific claims earn their keep? a claim can be perfectly retrievable and still be dead weight, or actively misleading. today nothing correlates "artifact X was surfaced into a session" with what happened in that session, so a reviewer deciding what toexpireis flying blind.this proposes a read-only, measurement-only effectiveness signal: per approved artifact, the association between it being in a session's context pack and a coarse session outcome, reported with a confidence interval and a deliberately conservative verdict.
proposed surface
state.db), and a per-session outcome signal derived from the append-only audit log (audit.read_events) — e.g. a session that ended withconfirm/citeactivity vs. one that ended incontradict/reject.useful/harmfulonly when the interval clears baseline and the sample meets--min-samples; otherwiseunverified/insufficient. an untrustworthy number never renders as a confident verdict.expire.a
src/vouch/eval/effectiveness.pymodule composesmetrics.compute+audit.read_events+ the context-pack surfacing log. the surfacing log is a new derived table inindex_db.SCHEMA(a cache, rebuildable — the yaml stays source of truth), added toindex_db.reset.review gate & scope
nothing here writes or edits knowledge and it never mutates trust or lifecycle — it reads the audit log and a derived cache and reports. it explicitly does not auto-
expireor auto-approve on a bad verdict; the human at the gate decides. the outcome signal is derived from events that already exist (the audit stream is authoritative), so there's no new authoritative state. if exposed askb.effectiveness, it touches the four registration sites (server.py,jsonl_server.py,capabilities.py,cli.py) with a test attests/test_effectiveness.py.acceptance criteria
vouch health effectivenessruns read-only; writes no artifacts, no audit events, no proposals.--window.useful/harmfulare withheld until the interval clears baseline and--min-samplesis met (elseunverified/insufficient).index_db.SCHEMAand cleared inindex_db.reset.--format jsonis a documented, stable schema.tests/test_effectiveness.pycovers the insufficient-sample path, a clear-signal path with a fixed clock, and the read-only invariant.make checkgreen.related: #226 (recall-quality eval — this is the outcome layer above it), #318 (synthesis faithfulness eval), #101 / #164 (metrics / stats).