Summary
The per-PR eval gate replays committed cassettes against a baseline that was itself generated by replaying those same cassettes, so per-case diffs are ~0 by construction — the paired bootstrap mainly fires on scoring/runner code changes, not model-quality drift (drift detection is consciously offloaded to the nightly live-API job). Compounding it, the README headline number pass_rate_strict has no PR-level regression gate at all.
Evidence
Makefile — evals-baseline (:96) builds evals/baselines/main.json by replaying sentinel/evals/cassettes; evals (:88) + evals-compare (:99, compare-to-baseline) replay the same cassettes against that baseline.
.github/workflows/ci.yml:112-199 (evals-gate, SENTINEL_EVAL_CASSETTE_MODE: replay) vs nightly-evals.yml (--live, the actual drift sensor).
sentinel/evals/cli.py:973 — the regression verdict iterates only category_match, hypothesis_cosine, action_coverage, evidence_quality. pass_rate_strict lives under headline (report.py:78) and is never read by _compute_regression_verdict (cli.py:951-1050).
Notes
Cassette-replay determinism is a deliberate, documented design choice — this is not "replay is wrong." It's: (a) make the README's "blocks regressions in CI" honest about what the PR gate vs nightly job each catch, and (b) give pass_rate_strict an explicit regression gate.
Fix
- Add a
pass_rate_strict regression gate (nightly at minimum).
- Tighten README wording to distinguish per-PR (scoring-code / structural) from nightly (model drift).
Summary
The per-PR eval gate replays committed cassettes against a baseline that was itself generated by replaying those same cassettes, so per-case diffs are ~0 by construction — the paired bootstrap mainly fires on scoring/runner code changes, not model-quality drift (drift detection is consciously offloaded to the nightly live-API job). Compounding it, the README headline number
pass_rate_stricthas no PR-level regression gate at all.Evidence
Makefile—evals-baseline(:96) buildsevals/baselines/main.jsonby replayingsentinel/evals/cassettes;evals(:88) +evals-compare(:99,compare-to-baseline) replay the same cassettes against that baseline..github/workflows/ci.yml:112-199(evals-gate,SENTINEL_EVAL_CASSETTE_MODE: replay) vsnightly-evals.yml(--live, the actual drift sensor).sentinel/evals/cli.py:973— the regression verdict iterates onlycategory_match, hypothesis_cosine, action_coverage, evidence_quality.pass_rate_strictlives underheadline(report.py:78) and is never read by_compute_regression_verdict(cli.py:951-1050).Notes
Cassette-replay determinism is a deliberate, documented design choice — this is not "replay is wrong." It's: (a) make the README's "blocks regressions in CI" honest about what the PR gate vs nightly job each catch, and (b) give
pass_rate_strictan explicit regression gate.Fix
pass_rate_strictregression gate (nightly at minimum).