A from-scratch hybrid rule + LLM-as-judge evaluation harness for LLM prompts and models.
-blueviolet){kind=icon}
Define an eval suite in YAML, point it at any model or prompt, and get back a pass-rate / judge-score / latency / cost scorecard, a win/tie/lose comparison, judge-vs-rule Cohen's κ agreement, an HTML report, and a full JSONL audit log — with retry+backoff and response caching for reproducibility. It runs fully offline with a built-in mock provider (no API key needed), so the whole pipeline is always inspectable.
Built to make "we do LLM evals" concrete: the hard part of evaluation isn't calling a model, it's scoring the output reliably and knowing how much to trust your judge. promptlab does both.
- Rules alone are brittle (they can't grade open-ended answers); an LLM judge alone is unreliable (position bias, verbosity bias, and it sometimes disagrees with ground truth). promptlab runs both and reports their Cohen's kappa agreement, so you can see exactly when the cheap judge can be trusted.
- Every raw model call and every grader verdict is written to
results.jsonl— a real observability/audit trail. - A CI gate (
--min-pass-rate) turns a suite into a pass/fail check you can wire into CI.
pip install -e .
promptlab run suites/starter.yaml --provider mockThis runs the bundled ~25-task starter suite against two offline mock targets and writes report/report.html + report/results.jsonl. Re-running reproduces identical numbers from the cache.
promptlab — eval summary
┏━━━━━━━━━━━━━━━┳━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
┃ target ┃ n ┃ pass % ┃ judge/5 ┃ p50 ms ┃ p95 ms ┃ cost $ ┃ retries ┃
┡━━━━━━━━━━━━━━━╇━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│ mock-baseline │ 25 │ 76.0 │ 4.40 │ 131 │ 199 │ 0.0000 │ 0 │
│ mock-improved │ 25 │ 100.0 │ 5.00 │ 82 │ 164 │ 0.0000 │ 0 │
└───────────────┴────┴────────┴─────────┴────────┴────────┴────────┴─────────┘
judge↔rule agreement: Cohen's κ = 0.615 (raw 90.0% over n=10)
mock-baseline_vs_mock-improved: win 0 / tie 19 / lose 6
The κ tells a real story. On the starter suite the judge and rules agree 90% of the time (κ = 0.615, "substantial"). The single disagreement is instructive: on json-hard-1 the weak model emits {'name': 'David', 'age': 52} — semantically correct but single-quoted, so the strict json_valid rule fails it while the (semantic) judge passes it. That gap is exactly what kappa is for.
| Layer | What it does |
|---|---|
| Rule graders | exact_match, regex, contains/not_contains, json_valid, json_schema_valid, numeric_tolerance, max_latency_ms — deterministic, offline, free |
| LLM-as-judge | structured {score 1-5, pass, reason} with position-swap + reference-guided rubric modes to cut bias; temperature 0 |
| Runner | task × target loop, exponential-backoff retry, response caching, full JSONL audit log |
| Metrics | pass rate, per-category & per-grader breakdown, p50/p95 latency, token cost, win/tie/lose, judge-vs-rule Cohen's κ |
| Report | Rich terminal table + self-contained HTML (no external assets) |
| CI gate | --min-pass-rate 0.8 → nonzero exit if any target falls below |
The committed report/ numbers run against the offline mock provider by design — so the
results are deterministic, free, keyless, and reproducible in CI by anyone who clones the repo
(the eval infrastructure is the subject, not any one vendor's model). To evaluate a real
model, set a key and run the bundled real-model suite (graders there tolerate real-model phrasing
- use the LLM judge, since real models won't emit a bare
positive):
export ANTHROPIC_API_KEY=... # or OPENAI_API_KEY
promptlab run suites/real-example.yaml # two real Claude targets, judged by Claudesuites/real-example.yaml evaluates claude-haiku-4-5 vs claude-sonnet-4-6 on extraction /
sentiment / math / refusal-safety / instruction-following. Any suite is just data — point a
target's provider/model at anthropic or openai and the same harness runs.
A suite is just data:
targets:
- {name: claude, provider: anthropic, model: claude-haiku-4-5, system_prompt: "Answer concisely."}
tasks:
- id: json-1
category: json-extract
input: "Return JSON with name and age. Text: John is 30."
graders:
- {type: json_schema_valid, schema: {type: object, required: [name, age]}}
- {type: llm_judge, rubric: "Does the JSON capture name=John, age=30?", reference: '{"name":"John","age":30}'}promptlab/
config.py # pydantic suite/task/target/grader schema (YAML)
providers/ # mock (offline) · anthropic · openai
graders/rules.py # deterministic graders
graders/llm_judge.py # LLM-as-judge + position-swap / reference-guided
runner.py # retry/backoff, caching, JSONL audit log
metrics.py # pass rate, kappa, latency, win/tie/lose
report.py # rich table + HTML
suites/starter.yaml # ~25 hand-written tasks
tests/ # pytest (rule graders + CI gate + end-to-end)
report/ # committed real outputs (report.html, results.jsonl)
Stated plainly, because knowing a tool's limits is part of building it:
- Scale — the starter suite is ~25 hand-written tasks; the numbers are indicative, not a statistically-powered benchmark, and the judge-vs-rule κ is computed on the handful of judge-graded tasks. It's a harness, not a benchmark dataset.
- Committed numbers use the mock provider — a deterministic offline stand-in (chosen so the repo is reproducible/keyless in CI). Real-model numbers come from
suites/real-example.yamlwith a key; they aren't committed because they cost money and aren't reproducible by a cloner. - Single-package, not a service — no hosted UI, no DB, not a Promptfoo competitor. Cost/latency are as-measured on the configured endpoint, not vendor benchmarks.
- Next steps if taken further: dataset versioning + larger suites, more judge models with inter-judge agreement (not just judge-vs-rule), pairwise/Elo ranking across many targets, and a GitHub Action that posts the scorecard on PRs.
MIT