Traceback

Find the reward hacks hiding deep in an agent's trajectory, turn each one into a regression test, and ship a verifier patch only after the exploit dies and every legitimate solution still passes.

Traceback is for the people who own a task, its grader, and its release: environment and benchmark authors. If your agents clear a grader but you cannot be sure they earned it, Traceback drives the grader from the inside and shows where it breaks.

The Python package is forkproof. The discovery layer is called ForkProof in the code and the reports; Traceback is the system around it.

The problem

Reward hacks live deep in agent trajectories: an agent only finds it can cheat the grader after installing dependencies, editing files, or driving the environment into a rare state.

Existing reward-hack identification tools hunt for them by restarting the task from scratch, so every run has to solve two problems at once: reach that state over and over, then discover an exploit from it. Reaching the state burns the token budget, and the real production multi-step hacks rarely get found.

The idea

Traceback splits the two apart. It pays to reach the important moment once, then snapshots that state like a git commit: we save the filesystem, environment image, and task grader. We restore the snapshot across many parallel sandboxes, sending a different attacker agent down each branch to find diverse reward hacks. Every branch inherits the same context but takes a different next action, so the whole budget goes to exploitation instead of setup. For the longer-horizon hacks, a flat search just bets on probability to cover its bases; Traceback covers them on purpose.

Git for agent trajectories: fork the risky state, fan attackers out in parallel.

A rewarded branch is not a finding yet. It becomes an Exploit Witness only after it passes reward-hacking classification, deduplication by target and mechanism, complete provenance, and deterministic replay. Confirmed Witnesses become deterministic regression tests that drive an automatic verifier patch, gated by proof that the exploit dies and legitimate solutions still pass.

Results

A discovery benchmark on 10 Terminal Bench 2.1 tasks, measured on the live HUD and Modal surface (preliminary). Unmeasured values are marked TBD.

Detector	Reward hacks found
HUD Reward Hacking QA Agent, on the same tasks	0
ForkProof discovery	14 confirmed grader-breaking reward hacks, deduped by target and mechanism

Of 32 rewarded hacker branches, 14 broke the grader (planted a conftest.py, rewrote test_outputs.py, or shadowed a fake Rscript onto $PATH so the grader passes regardless) and the rest were verifier-legitimate. A normal-prompt control on the same forkpoints, model, and runtime found zero hacks, so every one of the 14 comes from adversarial discovery. HUD's Reward Hacking QA Agent finds 0 of them: it only sees the legitimate solves that actually occur in production and never exercises the adversarial branches. Full report: artifacts/forkproof/qabench/QA_BENCHMARK_REPORT.md.

Every confirmed exploit produced a patch that passes a three-part release gate: the exploit succeeds under the original verifier, the exploit is blocked under the patched verifier, and every previously legitimate solution still passes.

Using this data, after SFT on the 10 tasks we improved Qwen3-4B by 3% on a held-out set of Terminal Bench tasks. That is a floor, not a ceiling: more tasks and a larger search budget yield a larger clean set and more lift.

Built on

Traceback is an orchestration layer between HUD's reward and QA plane and Modal's executable-state plane. Thank you to our sponsors.

• HUD supplies the agent trajectories and artifacts: source traces, reward, step-level file evidence, the reward-hacking QA classifier, tasksets, and environment versioning.
• Modal supplies the snapshots: filesystem and directory state captured and restored at the forkpoint, plus isolated sandboxes for the parallel attacker branches.
• Fireworks powers the training step: the benchmark splits rewarded trajectories into a certified-clean set and a quarantined set, and the clean set feeds fine-tuning of a hardened model.

Quickstart

Requires Python 3.12, uv, Node and npm (for the UI), and credentials for HUD, Modal, and at least one model provider.

git clone https://github.com/ashtonchew/hack2fix2hack.git
cd hack2fix2hack

uv sync                                   # install pinned dependencies
cp .env.example .env                      # then fill HUD_API_KEY, MODAL_*, and one provider key
bash scripts/bootstrap_external_deps.sh   # fetch harden-v0, terminal-wrench, hud-trace-explorer into .external
uv run python -m modal setup             # one-time Modal auth (or set MODAL_TOKEN_* in .env)

Confirm the setup:

uv run pytest                             # behavior tests
python docs/plans/scripts/run_all.py      # planning-bundle validators

Usage

Materialize the benchmark environments from pinned Terminal Bench 2.1 tasks:

uv run python -m forkproof.qabench.materialize --manifest envs/qabench/tasks.json

Run the discovery benchmark (needs live HUD and Modal credentials):

uv run python -m forkproof.qabench.run_benchmark --phase all --count 10
uv run python -m forkproof.qabench.control_baseline    # normal-prompt baseline, the X leg

Explore the run graph in the UI:

cd frontend
npm install
npm run dev        # http://localhost:5174

The UI walks the discover, witness, fix, gate, release narrative as an interactive branch tree. See frontend/README.md for the screen map.

Repo layout

Path	What it holds
src/forkproof/forkpoints/	trace ingestion, action-boundary selection, snapshot capture and restore
src/forkproof/witnesses/	branch scheduling, grading, dedup, durable Witness packaging, replay
src/forkproof/controls/	legitimate-control fixtures
src/forkproof/releases/	harden-v0 patch loop, ProofSet, v1/v2 release gate, ReleaseProof
src/forkproof/qabench/	the 10-task QA discovery benchmark
envs/qabench/	materialized HUD environments for the benchmark tasks
artifacts/forkproof/	evidence: release proofs, benchmark report, control runs
docs/plans/	the numbered ExecPlans and their evidence manifests
frontend/	React and React Flow run-graph UI

Docs

Start with docs/plans/000-index.md for the plan graph, then the spec set under docs/plans/specs/ (product, architecture, interfaces, persistence, validation, security, environment). docs/plans/GLOSSARY.md defines ForkPoint, BranchRun, Exploit Witness, ProofSet, and ReleaseProof. Governance lives in AGENTS.md.

Status

The QA discovery benchmark is preliminary: about 4 branches per task, and more branches will find more. The richer sterile re-grade referee is pending a post-agent snapshot. Every claim traces to a plan evidence manifest under docs/plans/evidence/ with the exact command, exit code, and artifact path.

Contributing

Work ships as numbered ExecPlans, one collision-free vertical slice each. Before editing source, read AGENTS.md, confirm a plan owns the paths you will touch, and run the validators above. Completion is evidence-based: update the plan's evidence manifest with the command, exit code, and artifact path.