Add vendored agent-eval-harness scoring engine (eval harness, 1/3)#21
Open
AshwinUgale wants to merge 1 commit into
Open
Add vendored agent-eval-harness scoring engine (eval harness, 1/3)#21AshwinUgale wants to merge 1 commit into
AshwinUgale wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of three focused PRs adding an evaluation harness under
eval/(split per your review request): scorer (this) → engine → corpus.Adds only the scoring engine — a byte-identical vendored copy of agent-eval-harness's scorer. Nothing uses it yet; the runner and judges that call it come in PR 2.
eval/scoring/— 6 files:score.py— the grader (judges + thresholds + regression detection)agent_eval/— its 4 support modules (config,events,_bootstrap,__init__)PROVENANCE.md— source repo, version, commit, and "modifications: none"Why vendored, not a pip dependency (per our discussion): I traced
score.py's imports — the path we use needs onlyconfig.py+events.py(stdlib +pyyaml); the LLM / MLflow / builtin-judge machinery is all lazy-imported behind branches the deterministic path never hits. So it's a small, unmodified copy with no import rewiring, re-syncable from upstream viaPROVENANCE.md, and it avoids coupling to their Claude-Code-skill config schema.Review scope: it's a faithful copy, so the check is provenance (source/commit in
PROVENANCE.md), not a line-by-line review. Strictly additive — nothing outsideeval/scoring/.One heads-up for the reviewer:
score.pyruns judgecheck:snippets viaexec()— that's by design (how the scorer evaluates inline checks), and it only ever executes a committed, PR-reviewed config file. The trust boundary is documented in PR 2'seval/README.md.cc @csoceanu