Skip to content

Evals: per-PR gate is structurally near-circular and pass_rate_strict has no regression gate #68

Description

@JumpTechCode

Summary

The per-PR eval gate replays committed cassettes against a baseline that was itself generated by replaying those same cassettes, so per-case diffs are ~0 by construction — the paired bootstrap mainly fires on scoring/runner code changes, not model-quality drift (drift detection is consciously offloaded to the nightly live-API job). Compounding it, the README headline number pass_rate_strict has no PR-level regression gate at all.

Evidence

  • Makefileevals-baseline (:96) builds evals/baselines/main.json by replaying sentinel/evals/cassettes; evals (:88) + evals-compare (:99, compare-to-baseline) replay the same cassettes against that baseline.
  • .github/workflows/ci.yml:112-199 (evals-gate, SENTINEL_EVAL_CASSETTE_MODE: replay) vs nightly-evals.yml (--live, the actual drift sensor).
  • sentinel/evals/cli.py:973 — the regression verdict iterates only category_match, hypothesis_cosine, action_coverage, evidence_quality. pass_rate_strict lives under headline (report.py:78) and is never read by _compute_regression_verdict (cli.py:951-1050).

Notes

Cassette-replay determinism is a deliberate, documented design choice — this is not "replay is wrong." It's: (a) make the README's "blocks regressions in CI" honest about what the PR gate vs nightly job each catch, and (b) give pass_rate_strict an explicit regression gate.

Fix

  • Add a pass_rate_strict regression gate (nightly at minimum).
  • Tighten README wording to distinguish per-PR (scoring-code / structural) from nightly (model drift).

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingTest coverage gaps or improvements

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions