feat(tbench2_env): per-task sandbox image recipe + Daytona materialization#966
Closed
nblintao wants to merge 3 commits into
Closed
feat(tbench2_env): per-task sandbox image recipe + Daytona materialization#966nblintao wants to merge 3 commits into
nblintao wants to merge 3 commits into
Conversation
nblintao
force-pushed
the
tb2-per-task-sandbox-recipe
branch
from
July 15, 2026 03:30
f5446f6 to
3f96f51
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
nblintao
force-pushed
the
tb2-per-task-sandbox-recipe
branch
from
July 15, 2026 04:04
3f96f51 to
61cca6b
Compare
5 tasks
…ask env & workdir - _evaluate_task runs the task's official tests/test.sh (pinned uvx toolchain, /logs/verifier/reward.txt) when the task ships it; the bare pytest fallback (custom task dirs without the canonical harness) prefers uvx with a pinned pytest so it carries its own toolchain, and runs from the same workdir the agent worked in. - Pass timeouts through to every shell_exec call: camel's shell_exec has its own per-call default (20s) and ignores the constructor timeout, so agent commands and evaluations were silently truncated at 20s (circuit-fibsqrt's verifier declares 3600s and was cut mid-test). Agent execs use command_timeout_s (TB2_COMMAND_TIMEOUT_S-overridable); evaluate uses the task's own [verifier].timeout_sec budget. - Shell cwd resolves to the task image's own WORKDIR, parsed from the task's environment/Dockerfile (TB2_TASK_WORKDIR overrides): task images pin their own WORKDIR (fix-git: /app/personal-site, prove-plus-comm: /workspace) and both agents and oracle solutions assume commands run there. Falls back to the task source dir for plain local mode. - Drop install_dependencies=['pytest'] (evaluation carries its own pytest via uvx); with CAMEL_RUNTIME=true camel's .initial_env venv then stops hijacking PATH away from the task image's native python. - Serialize canonical evaluation under a lock so concurrent sessions on one server cannot clobber the official fixed verifier paths (/tests, /logs/verifier/reward.txt).
nblintao
force-pushed
the
tb2-per-task-sandbox-recipe
branch
2 times, most recently
from
July 16, 2026 20:04
eea0d44 to
867e879
Compare
nblintao
force-pushed
the
tb2-per-task-sandbox-recipe
branch
2 times, most recently
from
July 16, 2026 21:00
1096c41 to
75785af
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 5 total unresolved issues (including 3 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 75785af. Configure here.
| ) | ||
| self._exec_in_container("rm -rf /task/tests && mkdir -p /task/tests") | ||
| self._copy_dir_to_container(tests_src, "/task/tests") | ||
|
|
There was a problem hiding this comment.
Docker evaluate skips test.sh
Medium Severity
This change adds canonical scoring via tests/test.sh and staged /tests for local mode, but Tbench2DockerEnvironment._evaluate_docker still runs bare python -m pytest under /task/tests, skipping the official harness and uvx-pinned toolchain official TB2 tasks require.
Reviewed by Cursor Bugbot for commit 75785af. Configure here.
Author
There was a problem hiding this comment.
Out of scope for this PR. Docker mode's bare-pytest scoring is pre-existing behavior; canonical tests/test.sh fidelity for docker mode is a separate track. What #972 does change in docker mode is stage-at-verify (tests excluded from the initial copy, injected only at verify), which is the reward-hacking fix. Canonical scoring there can follow up.
nblintao
force-pushed
the
tb2-per-task-sandbox-recipe
branch
3 times, most recently
from
July 16, 2026 21:46
ac56667 to
ecc6211
Compare
Both execution modes let the agent read (and tamper with) the assets the verifier scores against — tests/test_outputs.py spells out the expected outputs and solution/ is the literal answer. Fine for eval plumbing with a cooperative agent; for RL training it is a reward-hacking vector. Align with the official TB2 harness's model instead: the solution never ships, and tests exist in the agent's filesystem only during the verify window. - Local mode: with TB2_WITHHOLD_TESTS=1, reset() reads tests/ into server memory and deletes it (and solution/ — nothing in this env reads it; it exists only for oracle runs) from disk before the agent's first action. Gated off by default because in plain local mode TB2_TASKS_DIR may be a developer's working checkout; deployments that stage per-task copies (e.g. cloud sandboxes) should turn it on. The in-memory copy is process-local, so withhold mode assumes one server process owns the task filesystem for its lifetime (the per-task-sandbox deployment: one worker, one ephemeral sandbox per episode) — documented on the cache. - Evaluation stages a pristine tests/ copy at /tests for exactly the verify window — from server memory when withheld, i.e. a copy the agent never had filesystem access to — and removes it again once scoring returns, so a later episode / concurrent session sharing the filesystem can't read it. Both fixed paths (/tests, /logs/verifier) are recreated from scratch first, symlink included: the previous "mkdir -p && cp -a" merged into whatever already existed, and shutil.rmtree silently no-ops on a symlink, so an agent could pre-plant a /tests/conftest.py (pytest auto-loads it) or symlink a fixed path at a dir it controls and survive into scoring. _hard_remove unlinks the symlink itself. The in-memory tarball is restored with the 3.12 data filter when available, falling back cleanly on 3.10/3.11 (the package targets >=3.10). The bare-pytest fallback runs against the same staged copy. The canonical harness writes its test.sh stdout under /logs/verifier (not a fixed /tmp path) and the verify-window cleanup wipes /logs/verifier alongside /tests, so neither reward.txt nor the pytest -rA log — which can spell out expected values — outlives scoring for a timed-out episode or a later session. The withhold cache tars the realpath of tests/ so a symlinked tests/ caches its target's files rather than a bare link that would restore empty. - Docker mode gets true stage-at-verify: the initial /task copy excludes tests/ and solution/, and _evaluate_docker streams tests in from the host (this server sits outside the container) right before pytest runs, then removes them after scoring — in a finally, because docker-mode step() reports a scoring error without ending the episode, and the agent must not keep its session alongside a staged /task/tests that an errored verify left behind. Residual risk, shared with the official TB2 harness: verification necessarily runs inside the same container the agent controlled, and agent-started background processes must be left alive (many tasks are scored on services the agent set up), so a leftover watcher that copies the staged tests or rewrites the reward file during the verify window — or a root agent that tampers with container binaries — could still fake a pass. Task state (services, git state) is not portable, so verification cannot move to a container the agent never touched. tests/envs/test_tbench2_env.py covers the withhold round-trip, staging hygiene (pre-planted conftest.py and stale reward.txt wiped; symlinked fixed paths and a symlinked tests/ removed without following into the target; source checkout untouched without the gate), post-scoring cleanup, the full evaluate flow from the in-memory copy, and docker mode's copy exclusions, evaluate-time staging order, and cleanup when scoring itself raises (mock container — no docker needed). Also verified end-to-end on a per-task Daytona cloud sandbox (fix-git, gate enabled): the staged task dir carries no tests/solution, a filesystem-wide find turns up neither, pre-solution evaluate scores 0.0 via the canonical test.sh from the in-memory copy, and applying the oracle solution scores 1.0. A static scan of all 89 official tasks' tests/ trees (regular files and dirs only — no symlinks or special modes) confirms the tar round-trip is safe suite-wide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ation TB2 is per-task-image (task.toml [environment].docker_image pins the official image the TB2 harness itself runs); a cloud sandbox serving this env must be built per task: official task image + this env's server layer, one layer, no DinD. - Provider-agnostic core: _server_layer_commands is plain shell; this checkout's env source is embedded into the build so local fixes ship without a released package. The task image's apt index is preserved (solutions/agents run bare 'apt install' against it). - Daytona materialization: create_task_sandbox (declarative per-episode create, no named-snapshot quota, repeat creates hit the build cache; every task dir is staged from the checkout's pinned-SHA GitHub tarball — one uniform, payload-free path; sandboxes carry an ownership label for safe cleanup) plus an optional named-snapshot bake CLI as a warm cache. make_daytona is public API: callers driving create_task_sandbox need a client configured the same way this module's own CLI is. - Verifier-asset hygiene: the staged task directory excludes solution/ at build time (tar --exclude anchored to the one task), and SERVER_CMD sets TB2_WITHHOLD_TESTS=1 so the server withholds tests/ at reset and stages them only at verify (the underlying env fix). tests/envs/test_tbench2_task_snapshots.py pins both. Validated by a full 89-task golden sweep (official solutions + official test.sh scoring): 0 infra errors; failures traced to upstream solution drift, not the environment. The verifier-asset hygiene was re-verified end-to-end on a per-task Daytona sandbox (fix-git): no tests/solution anywhere agent-readable, pre-solution evaluate 0.0, oracle solution 1.0.
nblintao
force-pushed
the
tb2-per-task-sandbox-recipe
branch
from
July 16, 2026 21:58
ecc6211 to
eea94f0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Terminal-Bench-2 is a per-task-image benchmark: every task pins its official runtime image in
task.toml([environment].docker_image— the exact image the TB2 harness itself runs). This addstbench2_env/task_snapshots.py: a recipe that builds, per task, the official task image ⊕ this env's server layer (one layer, no docker-in-docker), plus a Daytona materialization that creates one such sandbox per episode. It lets any cloud-sandbox deployment serve TB2 with full per-task environment fidelity, instead of running every task inside one generic server container.Design notes:
_server_layer_commands()is plain shell — a uv-managed Python venv at/opt/envserverrunning this checkout'stbench2_envserver, and the task directory staged from the tasks checkout's pinned-commit GitHub tarball. The same commands can back a Dockerfile or another provider's build.create_task_sandbox()creates declaratively from theImagedefinition — no named snapshot, so no org snapshot quota; repeat creates hit Daytona's build cache (~1min after the first build). An optional bake CLI pre-registers named snapshots as a warm cache. Sandboxes carry an ownership label (openenv-tbench2-task) so cleanup tooling never touches other workloads in a shared org.apt installagainst it), uv is installed outside PATH (/opt/uv) so no tool the image didn't ship resolves from PATH — this matters in practice:financial-document-processor's official solution uses the image's own uv 0.8.14, and shadowing it with a newer uv broke its dependency resolution.solution/at build time (tar --excludeanchored to the one task — nothing in this env reads it), and the server command setsTB2_WITHHOLD_TESTS=1so tests are withheld into server memory atreset()and exist in the agent's filesystem only during the verify window, like the official harness.tests/envs/test_tbench2_task_snapshots.pypins both.tbench2_envsource so server fixes ship without a released package; once released upstream, this should switch to installing the published package.Stacked on #972, which stacks on #965. This branch is based on #972's, so both underlying
fixcommits appear in this PR until they merge — review this PR's own commit,feat(tbench2_env): per-task sandbox image recipe + Daytona materialization(the last commit;task_snapshots.pyplus its test file). Both fixes are functional prerequisites (the embedded server source needs them for per-task containers to score faithfully and hack-resistantly), so they should merge first; this PR then rebases to thefeatcommit alone. (GitHub can't retarget a fork PR's base onto another fork branch, so the base still readsmain.)Type of Change
Alignment Checklist
.claude/docs/PRINCIPLES.mdand this PR aligns with our principles (container isolation; one env = one trajectory — one sandbox per episode strengthens both).claude/docs/INVARIANTS.mdand no invariants are violated (no API changes; the module never imports fromserver/)bash .claude/hooks/lint.sh— the new files are clean (pre-existing failures in other envs are untouched)RFC Status
Env-level tooling with no core-abstraction or API change; happy to write an RFC if maintainers prefer one for the per-task-image pattern (it likely generalizes to other per-task/per-instance-image benchmarks, e.g. SWE-bench-style suites).
Test Plan
Validated end-to-end on Daytona against all 89 official TB2 tasks (with #965 applied), executing each task's official solution under the official oracle convention and scoring with the official
tests/test.sh:fix-git, with fix(tbench2_env): keep verifier assets out of the agent's reach #972's gate enabled by this recipe): notests/orsolution/anywhere agent-readable (filesystem-widefind), pre-solution evaluate 0.0 via the canonical harness from the in-memory copy, official oracle solution 1.0.tests/envs/test_tbench2_task_snapshots.py): the tar exclusion is anchored to the one task'ssolution/, and the server command carries the withhold gate.Note
Medium Risk
Changes reward computation, deletes or withholds verifier files on disk, and introduces cloud image builds; mitigated by locks, cleanup, and broad unit tests, but scoring and RL anti-cheat behavior are security-sensitive.
Overview
Adds
task_snapshots.pyto build official TB2 task images plus an OpenEnv server layer (uv at/opt/envserver, task tree from a pinned GitHub tarball at/opt/tb2-tasks), with Daytonacreate_task_sandbox/ optional snapshot bake CLI. Sandbox startup setsTB2_WITHHOLD_TESTS=1,CAMEL_RUNTIME=true, and per-taskTB2_DEFAULT_TASK_ID.Local and Docker env behavior is tightened for faithful, hack-resistant scoring: optional
TB2_WITHHOLD_TESTSremovestests/andsolution/from the agent filesystem atreset()(tests cached in server memory), stages a clean copy at/testsonly duringevaluate, and prefers the task’stests/test.shharness withtask.tomlverifier timeouts. Agents run in the task image WORKDIR (from Dockerfile or override), andshell_execgets explicit timeouts viaTB2_COMMAND_TIMEOUT_S. Docker mode skips copying verifier assets into/taskand stages tests only at verify time.Docs add
TB2_WITHHOLD_TESTS; tests cover withholding, symlink hardening, canonical evaluate, Docker staging, and snapshotsolution/exclusion.Reviewed by Cursor Bugbot for commit eea94f0. Bugbot is set up for automated code reviews on this repo. Configure here.