ProdSecBench

ProdSecBench is an open-source benchmark harness for evaluating large language models on realistic application security, product security, supply-chain security, and LLM application security work.

The benchmark is intentionally not a CTF. It simulates the security review desk: diffs, code snippets, design notes, dependency changes, messy product context, missing evidence, release decisions, and the need to be useful without hallucinating.

What this MVP includes

• Python CLI: prodsecbench
• Pydantic response/task models
• Public smoke and public-dev task suites
• Provider runners for echo, OpenAI-compatible APIs, OpenRouter, and Anthropic
• Suite run harness with JSON result artifacts
• Deterministic scorer with evidence-grounding penalties
• Canary extraction and recall probe generation
• Deterministic synthetic task generators
• Fixture-based semantic judge calibration
• Passphrase-protected sealed suite pack/unpack/verify infrastructure
• JSON response and task schemas
• CI workflow
• Detailed docs for methodology, scoring, contamination, task authoring, sealed suites, generators, providers, governance, and roadmap

Install locally

python -m venv .venv
. .venv/bin/activate
pip install -e '.[dev]'
pytest -q

Quick start

List tasks:

prodsecbench list-tasks tasks/public-smoke

Render a task:

prodsecbench show-task tasks/public-smoke/authz-idor-001/task.yaml

Score a model response:

prodsecbench score tasks/public-smoke/authz-idor-001/task.yaml response.json

Generate contamination probes from embedded canaries:

prodsecbench canary-probes tasks/public-smoke/authz-idor-001

Benchmark tracks

MVP track priorities:

• PR / diff security review: realistic code review with release decisions.
• AuthN/AuthZ reasoning: tenant isolation, object ownership, role/scope boundaries.
• Dependency / supply-chain triage: package updates, registry metadata, suspicious diffs, transitive impact.
• Threat modeling from product docs: assets, trust boundaries, abuse cases, requirements.
• LLM application security: prompt injection, tool abuse, RAG/data-boundary failures.

Planned tracks:

• Incident/vulnerability intake triage
• Secure design review
• Security requirements from ambiguous product tickets

Leaderboard

No official leaderboard is published yet. The MVP establishes the harness and public smoke suite first.

Planned leaderboard profiles:

• Product Security Engineer: PR review, authz, threat modeling, secure design, vuln triage.
• Supply Chain Security: dependency triage, transitive blast radius, registry/package evidence.
• LLM AppSec: prompt injection, tool permissions, RAG boundaries, data exfiltration.
• Engineering Usefulness: actionable fixes, low false positives, evidence grounding, concise output.

Future leaderboard entries will include:

• model/provider/version
• benchmark suite version
• evaluation date
• per-track scores
• hallucination rate
• false-positive rate
• missed release blockers
• contamination-risk signals
• cost/latency/context usage where available

Official leaderboard scores should come from sealed and fresh-generated suites, not public development tasks.

Public vs sealed suites

ProdSecBench separates openness from leaderboard integrity:

• public-smoke: tiny tasks for harness verification.
• public-dev: community regression and local testing; assumed contaminated over time.
• sealed-official: active leaderboard tasks, not committed as raw public artifacts.
• fresh-canary: newly generated anti-contamination tasks used for official runs.

See docs/CONTAMINATION.md for the full strategy.

Defensive scope

ProdSecBench is defensive and product-security-oriented. Tasks ask for findings, remediation, tests, missing evidence, and release decisions. They should not ask models to perform live exploitation, credential theft, malware generation, persistence, evasion, or real-target attack workflows.

Documentation

• docs/METHODOLOGY.md: benchmark philosophy and track design.
• docs/TRACKS.md: track-specific artifacts, outputs, false positives, and severity guidance.
• docs/PROVIDER_RUNNERS.md: model provider runner configuration and run artifacts.
• docs/GENERATORS.md: deterministic synthetic task generation.
• docs/SCORING.md: deterministic scoring, hallucination penalties, future judge model boundaries.
• docs/SEMANTIC_JUDGING.md: bounded semantic judge and calibration workflow.
• docs/RESPONSE_FORMAT.md: model response JSON contract.
• docs/CONTAMINATION.md: canaries, probes, public/private split, fresh generation.
• docs/SUITES.md: public, sealed, fresh, and retired suite operations.
• docs/SEALED_SUITES.md: sealed pack workflow and operational cautions.
• docs/TASK_AUTHORING.md: how to create realistic benchmark tasks.
• docs/LEADERBOARD.md: official leaderboard profile and run protocol.
• docs/REPRODUCIBILITY.md: prompt, model, run, retry, and provenance rules.
• docs/SAFETY.md: defensive boundary and examples.
• docs/GOVERNANCE.md: review, release, and sealed-suite process.
• docs/ROADMAP.md: MVP to v1 path.

License

Code is licensed under Apache-2.0. Public benchmark examples are intended for open use, but active official/sealed evals should remain unpublished until retired.