fix(tbench2_env): canonical test.sh scoring, real timeouts, task-image workdir

[nblintao]

Summary

Makes local-mode Tbench2Environment faithful to real Terminal-Bench-2 semantics: evaluate scores with each task's official tests/test.sh (pinned uvx toolchain, /logs/verifier/reward.txt), command execution honors real time budgets, and commands run in the task image's own WORKDIR. Follow-up to #503, which fixed evaluate always returning 0 — this makes the returned reward match the official TB2 verifier.

Details:

• Canonical scoring: bare pytest tests/ (current behavior) skips the toolchain the official harness pins (uvx: Python 3.13, pytest 8.4.1, ctrf) and mis-scores tasks that depend on it. _evaluate_task now runs tests/test.sh when the task ships it; the bare-pytest fallback remains for custom task dirs, upgraded to prefer uvx --with pytest==8.4.1 so it carries its own toolchain too.
• Real timeouts: camel's shell_exec has a per-call default of 20s and ignores the constructor timeout, silently truncating agent commands and evaluations (e.g. circuit-fibsqrt declares a 3600s verifier budget and was cut mid-test, scoring a false 0). Agent execs now pass command_timeout_s (overridable via TB2_COMMAND_TIMEOUT_S); evaluate passes the task's own [verifier].timeout_sec.
• Task workdir: task images pin their own WORKDIR and both agents and oracle solutions assume commands run there — it is not always /app (fix-git: /app/personal-site, prove-plus-comm: /workspace). The shell cwd now resolves from the task's environment/Dockerfile (TB2_TASK_WORKDIR overrides), falling back to the task source dir for plain local mode.
• Drop install_dependencies=['pytest']: evaluation now carries its own pytest via uvx; when the server runs with CAMEL_RUNTIME=true, camel's .initial_env venv then also stops shadowing the task image's native python on PATH.

Type of Change

• Bug fix

Alignment Checklist

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles (notably: rewards are computed inside the environment, per RFC 002)
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated (Gym API signatures unchanged; server-internal changes only)
• I have run bash .claude/hooks/lint.sh — the touched file is clean (pre-existing failures in other envs are untouched)

RFC Status

• Not required (bug fix, docs, minor refactoring)

Test Plan

Validated on real per-task TB2 containers (official task images), scoring official solutions under the official oracle convention:

• Full 89-task golden sweep: 82/89 score 1.0 with 0 infra errors; the 7 remaining failures trace to upstream solution drift in terminal-bench-2 itself (apt version pins gone from Debian, missing setuptools for pystan, etc. — happy to share the per-task evidence).
• Each fix maps to tasks that flip from a false 0.0 to 1.0: circuit-fibsqrt/compile-compcert (timeouts), fix-git/prove-plus-comm (workdir), plus correct scoring across the suite via test.sh.
• 11-task concurrent regression on the final revision: 11/11 pass.

Note: the canonical eval uses the official global paths (/tests, /logs/verifier/reward.txt), which assumes one concurrent session per container — consistent with TB2's one-task-per-container model (and with one-env-one-trajectory); flagging for reviewers running multi-session servers against a single container.

Claude Code Review

/alignment-review run on this change: Tier 1 (format) fixed; flags addressed during review — fallback now carries its own pytest toolchain; workdir resolution centralized server-side so all deployment modes agree. No principle or RFC conflicts identified.

---

Note

Medium Risk
Changes reward computation and execution semantics for TB2 local mode (timeouts, cwd, official harness); incorrect harness paths or multi-session-on-one-container setups could still mis-score, but Gym API surface is unchanged.

Overview
Aligns local Tbench2Environment with official Terminal-Bench 2 verifier behavior so evaluate rewards match real TB2 runs.

Evaluation now runs each task’s tests/test.sh when present (stages to /tests, reads /logs/verifier/reward.txt), with a thread lock so concurrent sessions don’t clash on shared harness paths. Tasks without test.sh still use a pytest fallback that prefers uvx --with pytest==8.4.1 and uses the agent workdir; verifier runs use [verifier].timeout_sec from task.toml.

Shell cwd is resolved from the task environment/Dockerfile WORKDIR (TB2_TASK_WORKDIR override, else task dir), and shell_exec passes command_timeout_s explicitly (configurable via TB2_COMMAND_TIMEOUT_S). install_dependencies=['pytest'] is removed so scoring doesn’t rely on the toolkit venv.

^{Reviewed by Cursor Bugbot for commit 501a20c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 289b100. Configure here.}

[cursor]

Wrong cwd when WORKDIR exists

High Severity

During reset, the parsed task WORKDIR is kept whenever Path(workdir).is_dir() is true. On the OpenEnv server image, /app already exists but is not the task tree, so many tasks whose Dockerfile uses WORKDIR /app run agent and verifier commands in the server root instead of the resolved task directory.

 

^{Reviewed by Cursor Bugbot for commit 289b100. Configure here.}