Scale skill evals: compare mode, per-prompt budgets, efficiency grader (uv + shared lib)#44

Open

Bwvolleyball wants to merge 57 commits into

mainfrom

feat/scale-skill-evals

Commits on May 29, 2026

build: bootstrap uv project for evals lib

Bwvolleyball
and
claude
committed
feat(evals): Pydantic data models with strict validation

Bwvolleyball
and
claude
committed
feat(evals): skill config loader with validation

Bwvolleyball
and
claude
committed
feat(evals): grading with per-prompt expected + budget verdict

Bwvolleyball
and
claude
committed
fix(evals): grading robustness from code review (loud on malformed checks, +tests)

Bwvolleyball
and
claude
committed
feat(evals): Harness protocol + claude-code adapter

Bwvolleyball
and
claude
committed
chore(evals): add one-time prompts CSV->YAML migration script

Bwvolleyball
and
claude
committed
feat(evals): migrate hawkscan prompts to YAML + worked budget example

Bwvolleyball
and
claude
committed
feat(evals): replay-from-trace regrade + realistic hw-07 fixture

Bwvolleyball
and
claude
committed
feat(evals): with/without-skill compare mode

Bwvolleyball
and
claude
committed
feat(evals): unified CLI (evals/compare/regrade/validate) + reporting

Bwvolleyball
and
claude
committed
feat(evals): migrate api prompts to YAML; remove CSV sources

Bwvolleyball
and
claude
committed
refactor(evals): harness scripts become shims into unified CLI

Bwvolleyball
and
claude
committed
ci(evals): tiered runs (validate on PR, cheap PR matrix, full main) + uv

Bwvolleyball
and
claude
committed
docs(evals): document uv CLI, prompts.yaml, compare/regrade, PASS-SLOW

Bwvolleyball
and
claude
committed
fix(evals): gate process checks on should_trigger+did_trigger (C1); drop unused --rubric

Bwvolleyball
and
claude
committed
ci+docs(evals): dispatch-only non-claude jobs (C2), add pytest job, drop rubric plumbing, refresh stale docs

Bwvolleyball
and
claude
committed
feat(evals): real codex adapter (ports pre-shim stream parsing)

Bwvolleyball
and
claude
committed
feat(evals): real cursor adapter (ports pre-shim stream parsing)

Bwvolleyball
and
claude
committed
fix(evals): cursor adapter launch restores --trust + skill-loading (live-run fidelity)

Bwvolleyball
and
claude
committed
feat(evals): real agy adapter (plain-text parsing)

Bwvolleyball
and
claude
committed
fix(evals): agy adapter appends OBSERVE_SUFFIX so triggers detect (live-run fidelity)

Bwvolleyball
and
claude
committed
ci(evals): re-enable codex/cursor/agy now that adapters exist (closes C2)

Bwvolleyball
and
claude
committed
feat(evals): CellReport model + cell.json artifact from main()

Bwvolleyball
and
claude
committed
feat(evals): render_job_summary (JUnit-style, failures first) + shields badge

Bwvolleyball
and
claude
committed
feat(evals): main() writes GITHUB_STEP_SUMMARY job report

Bwvolleyball
and
claude
committed
feat(evals): render_digest + report CLI; comment job posts rich digest

Bwvolleyball
and
claude
committed
feat(evals): baseline diff + score_delta (pure threshold math, no AI)

Bwvolleyball
and
claude
committed
feat(evals): digest shows regression vs released-tag baseline

Bwvolleyball
and
claude
committed
ci(evals): capture baseline at release tag; PR diffs against it (graceful)

Bwvolleyball
and
claude
committed
feat(evals): compare emits lift effect + writes lift.json

Bwvolleyball
and
claude
committed
feat(evals): render skill-lift section; PR runs compare for lift

Bwvolleyball
and
claude
committed
docs(evals): document JUnit-style report, comparisons, four real adapters

Bwvolleyball
and
claude
committed
fix(evals): comment job checkout-before-download (empty-digest bug); cursor best-effort; wire score_delta into overview

Bwvolleyball
and
claude
committed

Commits on Jun 1, 2026

Commits on Jun 2, 2026