feat(eval): add kb effectiveness outcome scoring

[dale053]

What changed

Adds a read-only, measurement-only effectiveness signal that correlates context-pack surfacing with session outcomes.
A new src/vouch/eval/effectiveness.py composes audit.read_events, metrics windowing, and a new derived context-pack surfacing cache.
Per approved artifact, it computes surfaced-good vs. surfaced-bad session counts, an associational lift over baseline, and a 95% Wilson confidence interval.
Verdicts are gated by statistical power (useful/harmful only when the interval clears baseline and --min-samples is met, else unverified/insufficient).
Artifacts are ranked by earned value.
The signal is surfaced as vouch health effectiveness [--window 90d] [--min-samples N] [--format text|json].
The same kb.effectiveness method is exposed over MCP/JSONL (registered across server.py, jsonl_server.py, capabilities.py, cli.py).
Surfacing is recorded from the read path (build_context_pack) into a new rebuildable context_surface table in index_db.SCHEMA, cleared by index_db.reset.

Why

Fixes #426.
vouch could measure retrieval quality (eval/recall.py) but not whether surfaced knowledge actually changes outcomes — so a reviewer deciding what to expire was flying blind.
This adds the outcome layer above recall: an honest per-artifact impact estimate that helps a human decide which approved claims to promote, re-cite, or expire, without ever making that call automatically.

What might break

No breaking changes for existing .vouch/ directories.
No artifact files move and no on-disk yaml field changes shape.
The only storage change is a new derived table (context_surface) in the state.db cache, which is auto-created and rebuildable — the yaml/audit log stay authoritative.
Existing kb.* methods are unchanged.
build_context_pack gains an optional session_id argument (defaults to None, so callers are unaffected).
The feature is purely additive and read-only: it writes no artifacts, no audit events, and no proposals.

VEP

Not a knowledge-surface change: no object-model, bundle-format, or audit-log shape change.
kb.effectiveness is a new read-only method and context_surface is a derived cache table (not authoritative state), matching the design already scoped in #426.
Flagging for maintainer confirmation that a VEP isn't required for the new read-only method + derived cache.

Tests

• make check passes locally (lint + mypy + pytest)
• New / changed behaviour has a test (tests/test_effectiveness.py: insufficient-sample path, clear-signal path with a fixed clock, read-only invariant, index_db.reset clears the surfacing table, CLI JSON output)
• CHANGELOG.md updated under ## [Unreleased]

[coderabbitai]

Warning

Review limit reached

@dale053, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 22 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f0203eee-08c3-4bcb-8699-f91e4f7b10d5

📥 Commits

Reviewing files that changed from the base of the PR and between 5c98a3a and ac838b0.

📒 Files selected for processing (12)

• CHANGELOG.md
• docs/effectiveness.md
• src/vouch/capabilities.py
• src/vouch/cli.py
• src/vouch/context.py
• src/vouch/eval/__init__.py
• src/vouch/eval/effectiveness.py
• src/vouch/index_db.py
• src/vouch/jsonl_server.py
• src/vouch/openclaw/context_engine.py
• src/vouch/server.py
• tests/test_effectiveness.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}