Skip to content

Add vendored agent-eval-harness scoring engine (eval harness, 1/3)#21

Open
AshwinUgale wants to merge 1 commit into
redhat-community-ai-tools:mainfrom
AshwinUgale:eval-scorer
Open

Add vendored agent-eval-harness scoring engine (eval harness, 1/3)#21
AshwinUgale wants to merge 1 commit into
redhat-community-ai-tools:mainfrom
AshwinUgale:eval-scorer

Conversation

@AshwinUgale

Copy link
Copy Markdown

First of three focused PRs adding an evaluation harness under eval/ (split per your review request): scorer (this) → engine → corpus.

Adds only the scoring engine — a byte-identical vendored copy of agent-eval-harness's scorer. Nothing uses it yet; the runner and judges that call it come in PR 2.

eval/scoring/ — 6 files:

  • score.py — the grader (judges + thresholds + regression detection)
  • agent_eval/ — its 4 support modules (config, events, _bootstrap, __init__)
  • PROVENANCE.md — source repo, version, commit, and "modifications: none"

Why vendored, not a pip dependency (per our discussion): I traced score.py's imports — the path we use needs only config.py + events.py (stdlib + pyyaml); the LLM / MLflow / builtin-judge machinery is all lazy-imported behind branches the deterministic path never hits. So it's a small, unmodified copy with no import rewiring, re-syncable from upstream via PROVENANCE.md, and it avoids coupling to their Claude-Code-skill config schema.

Review scope: it's a faithful copy, so the check is provenance (source/commit in PROVENANCE.md), not a line-by-line review. Strictly additive — nothing outside eval/scoring/.

One heads-up for the reviewer: score.py runs judge check: snippets via exec() — that's by design (how the scorer evaluates inline checks), and it only ever executes a committed, PR-reviewed config file. The trust boundary is documented in PR 2's eval/README.md.

cc @csoceanu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant