Skip to content

feat(evals): add capped live evaluation runner#54

Merged
ernestprovo23 merged 14 commits into
mainfrom
feat/h1-live-eval-runner
Jul 19, 2026
Merged

feat(evals): add capped live evaluation runner#54
ernestprovo23 merged 14 commits into
mainfrom
feat/h1-live-eval-runner

Conversation

@ernestprovo23

Copy link
Copy Markdown
Member

Summary

  • add a sequential, checkpointed paid-exploratory runner for all six frozen evaluation conditions
  • enforce an exact USD 10.00 ceiling with pessimistic reservations, authenticated checkpoints, one-process leasing, and bounded receipts
  • add a network-free worst-case estimator, guarded CLI execution, and sanitized deterministic replay fixtures
  • reserve native structured-output contracts, bound verdict-repair amplification, and reconcile all provider-reported token usage

Linear: DSE-708

Verification

  • 715 passed
  • ruff check .
  • ruff format --check .
  • git diff main...HEAD --check
  • gitleaks detect --no-banner --redact --source .
  • two read-only adversarial review passes; final verdict clean

No paid provider call was made before this PR. The live smoke remains a separate, explicitly capped correctness gate.

@ernestprovo23
ernestprovo23 merged commit 0e5f39b into main Jul 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant