feat(tbench2_env): orphan TTL + ownership labels for per-task sandboxes#991
Closed
nblintao wants to merge 4 commits into
Closed
feat(tbench2_env): orphan TTL + ownership labels for per-task sandboxes#991nblintao wants to merge 4 commits into
nblintao wants to merge 4 commits into
Conversation
…ask env & workdir - _evaluate_task runs the task's official tests/test.sh (pinned uvx toolchain, /logs/verifier/reward.txt) when the task ships it; the bare pytest fallback (custom task dirs without the canonical harness) prefers uvx with a pinned pytest so it carries its own toolchain, and runs from the same workdir the agent worked in. - Pass timeouts through to every shell_exec call: camel's shell_exec has its own per-call default (20s) and ignores the constructor timeout, so agent commands and evaluations were silently truncated at 20s (circuit-fibsqrt's verifier declares 3600s and was cut mid-test). Agent execs use command_timeout_s (TB2_COMMAND_TIMEOUT_S-overridable); evaluate uses the task's own [verifier].timeout_sec budget. - Shell cwd resolves to the task image's own WORKDIR, parsed from the task's environment/Dockerfile (TB2_TASK_WORKDIR overrides): task images pin their own WORKDIR (fix-git: /app/personal-site, prove-plus-comm: /workspace) and both agents and oracle solutions assume commands run there. Falls back to the task source dir for plain local mode. - Drop install_dependencies=['pytest'] (evaluation carries its own pytest via uvx); with CAMEL_RUNTIME=true camel's .initial_env venv then stops hijacking PATH away from the task image's native python. - Serialize canonical evaluation under a lock so concurrent sessions on one server cannot clobber the official fixed verifier paths (/tests, /logs/verifier/reward.txt).
Both execution modes let the agent read (and tamper with) the assets the verifier scores against — tests/test_outputs.py spells out the expected outputs and solution/ is the literal answer. Fine for eval plumbing with a cooperative agent; for RL training it is a reward-hacking vector. Align with the official TB2 harness's model instead: the solution never ships, and tests exist in the agent's filesystem only during the verify window. - Local mode: with TB2_WITHHOLD_TESTS=1, reset() reads tests/ into server memory and deletes it (and solution/ — nothing in this env reads it; it exists only for oracle runs) from disk before the agent's first action. Gated off by default because in plain local mode TB2_TASKS_DIR may be a developer's working checkout; deployments that stage per-task copies (e.g. cloud sandboxes) should turn it on. The in-memory copy is process-local, so withhold mode assumes one server process owns the task filesystem for its lifetime (the per-task-sandbox deployment: one worker, one ephemeral sandbox per episode) — documented on the cache. - Evaluation stages a pristine tests/ copy at /tests for exactly the verify window — from server memory when withheld, i.e. a copy the agent never had filesystem access to — and removes it again once scoring returns, so a later episode / concurrent session sharing the filesystem can't read it. Both fixed paths (/tests, /logs/verifier) are recreated from scratch first, symlink included: the previous "mkdir -p && cp -a" merged into whatever already existed, and shutil.rmtree silently no-ops on a symlink, so an agent could pre-plant a /tests/conftest.py (pytest auto-loads it) or symlink a fixed path at a dir it controls and survive into scoring. _hard_remove unlinks the symlink itself. The in-memory tarball is restored with the 3.12 data filter when available, falling back cleanly on 3.10/3.11 (the package targets >=3.10). The bare-pytest fallback runs against the same staged copy. The canonical harness writes its test.sh stdout under /logs/verifier (not a fixed /tmp path) and the verify-window cleanup wipes /logs/verifier alongside /tests, so neither reward.txt nor the pytest -rA log — which can spell out expected values — outlives scoring for a timed-out episode or a later session. The withhold cache tars the realpath of tests/ so a symlinked tests/ caches its target's files rather than a bare link that would restore empty. - Docker mode gets true stage-at-verify: the initial /task copy excludes tests/ and solution/, and _evaluate_docker streams tests in from the host (this server sits outside the container) right before pytest runs, then removes them after scoring — in a finally, because docker-mode step() reports a scoring error without ending the episode, and the agent must not keep its session alongside a staged /task/tests that an errored verify left behind. Residual risk, shared with the official TB2 harness: verification necessarily runs inside the same container the agent controlled, and agent-started background processes must be left alive (many tasks are scored on services the agent set up), so a leftover watcher that copies the staged tests or rewrites the reward file during the verify window — or a root agent that tampers with container binaries — could still fake a pass. Task state (services, git state) is not portable, so verification cannot move to a container the agent never touched. tests/envs/test_tbench2_env.py covers the withhold round-trip, staging hygiene (pre-planted conftest.py and stale reward.txt wiped; symlinked fixed paths and a symlinked tests/ removed without following into the target; source checkout untouched without the gate), post-scoring cleanup, the full evaluate flow from the in-memory copy, and docker mode's copy exclusions, evaluate-time staging order, and cleanup when scoring itself raises (mock container — no docker needed). Also verified end-to-end on a per-task Daytona cloud sandbox (fix-git, gate enabled): the staged task dir carries no tests/solution, a filesystem-wide find turns up neither, pre-solution evaluate scores 0.0 via the canonical test.sh from the in-memory copy, and applying the oracle solution scores 1.0. A static scan of all 89 official tasks' tests/ trees (regular files and dirs only — no symlinks or special modes) confirms the tar round-trip is safe suite-wide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ation TB2 is per-task-image (task.toml [environment].docker_image pins the official image the TB2 harness itself runs); a cloud sandbox serving this env must be built per task: official task image + this env's server layer, one layer, no DinD. - Provider-agnostic core: _server_layer_commands is plain shell; this checkout's env source is embedded into the build so local fixes ship without a released package. The task image's apt index is preserved (solutions/agents run bare 'apt install' against it). - Daytona materialization: create_task_sandbox (declarative per-episode create, no named-snapshot quota, repeat creates hit the build cache; every task dir is staged from the checkout's pinned-SHA GitHub tarball — one uniform, payload-free path; sandboxes carry an ownership label for safe cleanup) plus an optional named-snapshot bake CLI as a warm cache. make_daytona is public API: callers driving create_task_sandbox need a client configured the same way this module's own CLI is. - Verifier-asset hygiene: the staged task directory excludes solution/ at build time (tar --exclude anchored to the one task), and SERVER_CMD sets TB2_WITHHOLD_TESTS=1 so the server withholds tests/ at reset and stages them only at verify (the underlying env fix). tests/envs/test_tbench2_task_snapshots.py pins both. Validated by a full 89-task golden sweep (official solutions + official test.sh scoring): 0 infra errors; failures traced to upstream solution drift, not the environment. The verifier-asset hygiene was re-verified end-to-end on a per-task Daytona sandbox (fix-git): no tests/solution anywhere agent-readable, pre-solution evaluate 0.0, oracle solution 1.0.
A caller that dies without reaching its delete (SIGKILL, OOM, node loss) used to leak its sandbox forever: auto_stop_interval=0 means Daytona never reclaims it, and running orphans bill CPU+memory+disk indefinitely. Arm auto-stop (30 min) + auto-delete (120 min after stop) at create as a dead-man's switch. Daytona's auto-stop clock counts only SDK interactions — preview-proxy traffic, which is ALL of an episode's I/O, does not reset it — so a keepalive daemon thread beats refresh_activity() for as long as the creating process lives: a live episode of any length is safe, a dead caller stops beating and its orphan is reclaimed within the TTL. Also label each sandbox with who launched it (OPENENV_LAUNCHER or the unix user) and an optional run id (OPENENV_RUN_ID): the Daytona API records no creator, so in a shared org labels are the only way to attribute sandboxes and scope cleanup sweeps to one launcher or one run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cfe75d0. Configure here.
|
|
||
| threading.Thread( | ||
| target=_beat, name=f"tb2-sandbox-keepalive-{task_id}", daemon=True | ||
| ).start() |
There was a problem hiding this comment.
Keepalive thread exits permanently
Medium Severity
The sandbox keepalive daemon stops after three consecutive failed refresh_activity calls and never restarts, even when the launcher process is still running and the sandbox still exists. A Daytona API outage longer than the tolerated window therefore leaves long episodes without heartbeats, so auto-stop can terminate a healthy sandbox while the caller still believes the episode is active.
Reviewed by Cursor Bugbot for commit cfe75d0. Configure here.
nblintao
marked this pull request as draft
July 17, 2026 19:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A trainer/launcher that dies without reaching its
daytona.delete()(SIGKILL, OOM, node loss) leaks its sandbox forever:create_task_sandboxarmedauto_stop_interval=0, so Daytona never reclaims it, and a running orphan bills CPU+memory+disk indefinitely. We hit exactly this: a batch launch died ~1 minute in and left 30 orphans running for 25+ hours until swept by hand. In a shared org there was also no way to tell whose sandboxes they were — the Daytona API records no creator.Fix
Orphan TTL as a dead-man's switch. Create params now arm
auto_stop_interval=30+auto_delete_interval=120(both overridable kwargs). The subtlety: Daytona's auto-stop clock counts only SDK interactions — preview-proxy traffic does not reset it (docs), and an episode's I/O runs entirely over the signed preview URL, so a bare TTL would kill healthy episodes longer than the interval. A keepalive daemon thread therefore beatssandbox.refresh_activity()every 5 minutes for as long as the creating process lives:Ownership labels. Each sandbox now carries, alongside the existing
openenv-tbench2-task:openenv-launcher—$OPENENV_LAUNCHERif set (recommended on shared hosts where the unix user is a generic account), else the local unix useropenenv-run-id—$OPENENV_RUN_ID, when setso shared-org cleanup sweeps can target one launcher or one run (
daytona.list(labels={"openenv-launcher": ...})) instead of guessing.Testing
sandbox_labels()unit-checked: default (unix user), explicitOPENENV_LAUNCHER/OPENENV_RUN_ID, run-id omitted when unsetcreate_task_sandboxsignature is backward compatible (new params are keyword-only with defaults); the miles-side caller needs no change🤖 Generated with Claude Code
Note
Medium Risk
Changes how rewards are computed and when verifier files exist on disk (including destructive deletes when withholding is on), which affects training fairness and local checkouts; Daytona auto-stop plus keepalive changes operational behavior for long episodes and orphaned sandboxes.
Overview
This PR hardens Terminal-Bench 2 for RL and cloud sandboxes: agents no longer get durable access to
tests/orsolution/, and scoring aligns with the official harness.Local mode adds optional
TB2_WITHHOLD_TESTS(documented in the README): onreset(),tests/is tarballed into server memory and bothtests/andsolution/are removed from the task tree; evaluation stages a pristine copy at/testsonly inside a locked verify window, then wipes/testsand/logs/verifier. Staging uses_hard_removeso symlink tricks and plantedconftest.py/ fake rewards do not survive. Scoring prefers the task’stests/test.sh(reward from/logs/verifier/reward.txt) withtask.tomlverifier timeouts; fallback pytest usesuvxwhen available. Agent commands use the task image WORKDIR (from Dockerfile or env) and explicitTB2_COMMAND_TIMEOUT_Son eachshell_exec.Docker mode mirrors this: initial container copy excludes
testsandsolution, stages tests only atevaluate, and always removes staged/task/testsafterward (including on errors).task_snapshots.py(new) defines per-task official image + env server builds for Daytona, excludessolution/from staged tasks, starts the server withTB2_WITHHOLD_TESTS=1andCAMEL_RUNTIME=true, and addscreate_task_sandboxlifecycle: default auto-stop / auto-delete intervals,sandbox_labels(openenv-tbench2-task,openenv-launcher, optionalopenenv-run-id), and a daemon keepalive that callsrefresh_activity()so long preview-only episodes are not stopped while the launcher process is alive.Unit tests cover withholding, canonical evaluate, Docker copy/staging, and snapshot recipe details.
Reviewed by Cursor Bugbot for commit cfe75d0. Bugbot is set up for automated code reviews on this repo. Configure here.