openenv/tbench2: Daytona sandbox mode — one cloud sandbox per episode, as its own agent function by nblintao · Pull Request #1675 · radixark/miles

nblintao · 2026-07-14T23:18:49Z

Final PR of the stack (#1710 sandbox recipe → #1711 orphan TTL + ownership labels → this): the adapter-side Daytona sandbox mode. The existing shared-server path is behaviorally unchanged (asserted by a dispatch unit test).

What

A second agent function for the OpenEnv TB2 adapter: point --custom-agent-function-path at openenv_daytona_agent_function.run (the launcher does this automatically when OPENENV_TB2_TASKS_DIR is set) and every episode runs in its own Daytona cloud sandbox, built from the task's official image (task.toml docker_image) plus a tbench2_env server layer, deleted when the episode ends.

Same per-task image fidelity as docker mode, with zero shared infrastructure: no Docker socket, no env server to size or babysit (no MAX_CONCURRENT_ENVS / ulimit tuning), no cross-episode state. Cost profile: the first episode of a task builds its image (~10 min, layer-cached by definition hash), repeats start in ~1 min; no named snapshots, so no org snapshot quota. Sandboxes carry ownership labels and a TTL dead-man's switch (#1711), and creation is throttled process-wide with jittered backoff (Daytona rate-limits creates).

Architecture (shaped by review)

Two agent-function modules with the identical public shape — run() for miles (session-server policy wiring + training failure semantics: timeout → reward 0, no verdict → drop sample) and run_episode() for callers that bring their own policy client and failure semantics (eval_tbench2_via_api):

openenv_agent_function — the shared env server leg, and home of the shared agent loop + training wrapper.
openenv_daytona_agent_function — the Daytona sandbox mode; holds all Daytona-specific machinery (throttled creates, cancel-reap, per-episode env lifecycle).

Everything that differs between the legs enters the shared loop as three keyword parameters filled in only by each module's run_episode: run_body (how an env comes into being), native_evaluate (server contract, below), post_episode (shared-server trial-dir hygiene). No backend objects, no env-var-sniffing dispatch. Behavior change: setting OPENENV_TB2_TASKS_DIR while pointing at openenv_agent_function.run no longer switches modes implicitly — mode selection is explicit plugin selection.

Why the two legs score differently (`native_evaluate`)

Each leg matches what its env server actually provides:

Shared-server leg (unchanged): adapter-driven canonical exec + reward-marker parse, compensating for an UNMODIFIED older upstream tbench2_env server.
Daytona sandbox mode (new): the standard evaluate action. The recipe (openenv/tbench2: per-task sandbox image recipe + Daytona materialization #1710) bakes the installed tbench2_env into each task image; upstream main (≥ fix(tbench2_env): canonical test.sh scoring, real timeouts, task-image workdir huggingface/OpenEnv#965 + [fix] [test] attention_output_gate TP slice when num_kv_heads < TP #972) carries the server-side contract this leg scores through — canonical tests/test.sh run natively, task WORKDIR resolved server-side (WORKDIRs are not uniformly /app), and verifier assets withheld outside the verify window (TB2_WITHHOLD_TESTS, the reward-hacking fix). The launcher preflights the installed server source for exactly this contract: an older install wouldn't fail per-episode, it would silently mis-score every episode.

Once the shared-server deployment upgrades to current upstream, a follow-up can flip native_evaluate there and retire the compensation machinery.

Daytona key supply (review follow-up)

The key value never rides ray's logged channels (the submit command is echoed into driver logs; runtime_env persists in job metadata). Workers resolve DAYTONA_API_KEY from their own node-local environment (platform-injected, or single-host inheritance) first, else read a key file (DAYTONA_API_KEY_FILE, default ~/.config/daytona/api_key — a dotfile, K8s Secret mount, or shared-FS path). The launcher forwards only the file PATH and fails fast with provisioning guidance when neither supply is available. requirements.txt now also declares openai (already load-bearing in miles core; previously satisfied only transitively via sglang).

Tooling (ships with the mode; produced the validation evidence)

scan_golden.py — infra baseline with no LLM: replays each task's OFFICIAL solution/solve.sh (oracle-faithful, chunk-uploaded from the local checkout since the image withholds verifier assets) through the same sandbox + evaluate scoring; --logs captures solve/test log tails for anything below 1.0.
eval_tbench2_via_api.py — the exact training agent-env loop with any OpenAI-compatible API as the policy (no GPU, no Ray/Megatron), via each module's run_episode.
tests/ — 20 offline unit tests (pytest examples/experimental/openenv/tests/ -q): episode dispatch on both legs through the public run_episode entries, create-throttling retry/backoff/give-up + cancel-reap, typed throttle-error classification, key-resolution chain, recipe hygiene, labels, TTL keepalive lifecycle.

Validation

Post-refactor re-validation (4×H200 devbox, full stack green)

Re-ran the whole pipeline against the final two-module form to confirm the split (own agent function + native_evaluate params + no backend objects) broke nothing. tbench2_env installed from upstream OpenEnv main (39c91bf, the #965/#972 merge the Daytona leg scores through):

Offline — 20 passed (pytest examples/experimental/openenv/tests/ -q).
Daytona leg, no LLM (scan_golden.py) — golden 3/3 (fix-git, chess-best-move, circuit-fibsqrt all 1.0) through the real per-task-image sandbox + native evaluate.
Daytona leg, live API policy (eval_tbench2_via_api.py) — fix-git solved by DeepSeek-V4-Flash through the Daytona sandbox (dispatch confirmed env=daytona sandboxes).
Shared-server leg, real Docker fidelity — ran the shared leg (public run_episode, OPENENV_TB2_TASKS_DIR unset → dispatch selects openenv_agent_function, not the Daytona module) end-to-end against a live TB2_MODE=docker tbench2_env server (per-task official-image containers), with DeepSeek-V4-Flash as the policy: 1/2 tasks solved, 0 errored, reward correctly 1.0 (real tests/test.sh passed in the real container) vs 0.0 (real fail) — the native_evaluate=False canonical-exec scoring branch (byte-identical to pre-split, just relocated) produces correct rewards against a real Docker-mode server. Also covered by the dispatch unit test.
Full GRPO loop closure on 4×H200 (GLM-4.7-Flash colocated, TP=2 EP=2 DP=2 — the 4-GPU analog of the 8-GPU recipe; torch_dist checkpoint reshards automatically from its saved (TP1,PP8) layout): rollout 0 (2 fix-git episodes via Daytona sandboxes, raw_reward 0.5) → train_step outcome=NORMAL → weight update (461 tensors broadcast to the SGLang engines) → rollout 1 with updated weights (raw_reward 1.0) → train_step outcome=NORMAL. On-policy fidelity: rollout vs. ref logprobs -0.535/-0.543, KL ≈ 0.0115 (token-faithful TITO recording). Robustness confirmed under a saturated org: while the Daytona org CPU quota was fully consumed by an unrelated resident pool, the leg's DaytonaValidationError (Total CPU limit exceeded) was caught per-episode and the sample dropped cleanly across hundreds of failed creates — no crash, no leaked sandbox.

Prior evidence (larger scale, pre-split)

scan_golden.py — golden sweep 82/89 of the TB2 suite (0 infra errors; the 7 remaining have upstream-broken solutions, each with solve.log evidence).
eval_tbench2_via_api.py — full 89-task sweep with DeepSeek-V4-Flash (single sample, 30-turn cap, up to 38 concurrent episodes) — 33/89 solved, 0 systematic infra failures.
GRPO smoke on 8×H200 (TP=4 EP=2, 8-task subset): rollout 16/16 episodes through Daytona sandboxes, train step outcome=NORMAL, weight update, next rollout — loop closed; prefix-cache hit rate 0.96; sandboxes carried the TTL/ownership contract in production.

🤖 Generated with Claude Code

gemini-code-assist

Code Review

This pull request introduces support for per-task Daytona cloud sandboxes as an alternative to a shared environment server in the OpenEnv Terminal-Bench-2 adapter. It includes documentation updates, new standalone evaluation and validation scripts (eval_tbench2_via_api.py and scan_golden.py), and offline unit tests. The reviewer's feedback focuses on improving code robustness, such as replacing assert with ValueError for argument validation, using isinstance checks instead of parsing exception strings, and explicitly checking for environment variables rather than relying on exception handling.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Shi-Dong · 2026-07-22T12:43:47Z

+                "not upstream main"
+            )
+        env["OPENENV_TB2_TASKS_DIR"] = args.openenv_tb2_tasks_dir
+        env["DAYTONA_API_KEY"] = args.daytona_api_key


Hmm if we save DAYTONA_API_KEY here, the ray job might print it out in the logs, which means that the key might be leaked.

Good catch! Fixed. I changed it to read from server env or file directly, instead of passing through ray env.:

DAYTONA_API_KEY the Daytona API key, authenticating every sandbox create/delete. Read from the worker's own node-local environment; nothing forwards it. Supply it via platform-injected pod env, or by exporting it in the shell that starts ray on a single host. DAYTONA_API_KEY_FILE fallback when DAYTONA_API_KEY is unset: path of a file holding the key (default ~/.config/daytona/api_key). Launchers forward this path instead of the key itself, because ray runtime_env is logged in plaintext. Point it at a file every node can read: a dotfile, K8s Secret mount, or shared-FS path.

I noticed that wandb keys may have similar issue. Let me create a separate PR to fix that.

nblintao · 2026-07-23T01:50:05Z

 them with the GPU workload, run the env server on a separate Docker host and point
 the launcher at it via `--openenv-env-url http://<env-host>:8003`.

+### 2b. Alternative: Daytona cloud sandboxes (no Docker host)


I'll have another round of polish after some more factorings in a separate PR

nblintao · 2026-07-23T03:31:07Z

I've re-ran the validation. See the updated summary.

Shi-Dong · 2026-07-24T04:34:49Z

+    (e.g. "ThrottlerException: Too Many Requests").
+    """
+    try:
+        from daytona.common.errors import DaytonaRateLimitError


please hoist this import to the beginning of the script

I think it's better to keep this one in function because of SDK compatibility issue. I've updated the details in the docstring.

Shi-Dong · 2026-07-24T04:34:50Z

+
+
+def _start_declarative(task_id: str, tasks_dir: str) -> tuple[Any, str]:
+    import tb2_sandbox_daytona


please hoist this import to the beginning of the script

Shi-Dong · 2026-07-24T04:36:06Z

+                # [solution].env exported, cwd = task workdir.
+                sol_env = "DEBIAN_FRONTEND=noninteractive"
+                try:
+                    import tomllib


please hoist this import to the beginning of the script

Shi-Dong

LGTM with minor comment.

Shi-Dong · 2026-07-24T05:14:44Z

+            # exit-code marker to parse.
+            eval_result = await env.step(action_cls(action_type="evaluate"))
+            eval_time = time.monotonic() - t0
+            reward = float(getattr(eval_result, "reward", 0.0) or 0.0)


If there is an error in task eval, probably we should just leave reward = None? In the PR description it says "no verdict → drop sample", but it seems here that no verdict returns zero reward.

Ahh, good catch. Fixed.

Shi-Dong · 2026-07-24T12:32:26Z

+        else:
+            # Older server: adapter-driven canonical exec + marker parse.
+            eval_result = await env.step(action_cls(action_type="exec", command=_CANONICAL_EVAL_CMD))
+            eval_time = time.monotonic() - t0
+            eval_output = _obs_field(eval_result, "output")
+            reward = _parse_reward_marker(eval_output)
+            testsh_rc = _parse_testsh_rc(eval_output)


I think we can probably remove this branch since the changes have already landed in OpenEnv?

Yes. I‘ll remove that in a separate PR. Note the non-daytona path (the one you created) is still using this for now. In openenv, part of the fixes s only done on local mode. I need to fix docker too.

TB2 is a per-task-image benchmark: every task pins its official runtime image in task.toml, so a cloud sandbox serving the env must be built per task — the official task image plus a tbench2_env server layer, one layer, no DinD. This module owns that recipe, miles-side, next to its only consumer (the per-task Daytona backend of the openenv TB2 adapter). Provider-agnostic core: _server_layer_commands() emits plain shell commands (uv-managed venv at /opt/envserver running the installed tbench2_env package's source, embedded into the build; task dir staged from the tasks checkout's pinned-SHA GitHub tarball with solution/ excluded), so the same layers can back a Dockerfile or another provider. Daytona materialization: create_task_sandbox() creates per-episode declaratively from the Image definition — no named snapshots, no org snapshot quota; repeat creates hit Daytona's build cache — then execs server_cmd() and waits for /health. A bake CLI can pre-register named snapshots as a warm cache. The embedded env source is located from the installed tbench2_env package and requires an editable/checkout install (pyproject.toml must be present); _env_src_dir() fails fast with instructions instead of erroring mid-build. Nothing imports this module yet; the adapter's per-task backend adopts it in a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…half Review feedback on #1710: the module mixed two concerns its own docstring already kept apart. tb2_task_recipe.py now owns everything Daytona-agnostic (server layer commands, env-source embedding, task staging, server_cmd, task.toml reading, /health polling); tb2_task_sandbox.py keeps its name, CLI, and import surface but is purely the Daytona materialization (declarative Image build, Resources, create_task_sandbox, bake). The two symbols that crossed the new module boundary go public with the move: server_layer_commands and resolve_docker_image. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The tbench2_env fixes are upstream now (huggingface/OpenEnv#965 + #972); embedding the installed source remains the mechanism that guarantees the sandbox runs exactly the version validated locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review follow-ups on the per-task sandbox recipe: - _dir_tar_b64 is now byte-for-byte deterministic for identical source: gzip mtime=0 suppresses the header compression timestamp and _tar_filter zeroes per-entry mtimes/owners. The b64 is embedded in a build command, so the previous per-call drift changed the image definition on every episode create — defeating the provider build cache the declarative path relies on, and preventing pre-baked snapshots from ever matching. Guarded by a regression test. - rename the module pair to tb2_sandbox_recipe (provider-agnostic recipe) + tb2_sandbox_daytona (Daytona materialization): the subject of both is the sandbox, the trailing token is the role/provider, and future backends slot in as tb2_sandbox_<provider>. - drop the Daytona reference from _task_layer_command's docstring; the recipe module is provider-agnostic and the concrete ceiling is already documented at _MAX_INLINE_TAR_BYTES. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add a second execution backend to the OpenEnv TB2 adapter: instead of a shared env server on a Docker host, every episode can run in its OWN Daytona cloud sandbox, built from the task's official image (task.toml docker_image) plus a tbench2_env server layer, and deleted when the episode ends. Same per-task image fidelity as docker mode with zero resident infrastructure (no Docker socket, no server to size or babysit) and zero cross-episode state leakage. Backend selection is episode-scoped in _multi_turn: set OPENENV_TB2_TASKS_DIR to a terminal-bench-2 checkout and each episode declaratively builds its sandbox from the task's Image definition (layer-cached by definition hash: first episode of a task ~10 min, repeats ~1 min; no named snapshots, so no org snapshot quota; sandboxes carry an openenv-tbench2-task=<task_id> label for safe sweeping in a shared org). Unset -> the existing OPENENV_ENV_URL shared-server path, behaviorally unchanged (the shared leg now reads OPENENV_ENV_URL at its own use site instead of threading it through _multi_turn's signature). The two backends deliberately score differently, each matching what its server provides: the shared-server leg keeps the adapter-driven canonical exec + reward-marker parse (compensates for an UNMODIFIED upstream server); the per-task leg uses the standard evaluate action, because the sandbox recipe (tbench2_env.task_snapshots -- tbench2_env is OpenEnv's Terminal-Bench-2 env package, envs/tbench2_env in huggingface/openenv; fixes proposed upstream in openenv#965 + #966) bakes a patched server that runs the same canonical tests/test.sh natively and resolves the task WORKDIR server-side (so no _apply_workdir prefix on this leg either -- task WORKDIRs are not uniformly /app: fix-git, prove-plus-comm). Sandbox creation is throttled process-wide with jittered backoff (Daytona rate-limits creates). Ships with the two tools that produced the validation evidence, both driving the adapter's own code paths so what they check is what training runs: scan_golden.py -- infra baseline without any LLM: replay each task's OFFICIAL solution/solve.sh (oracle-faithful: staged at /solution, DEBIAN_FRONTEND=noninteractive, task.toml [solution].env exported, task workdir cwd) through the same sandbox + evaluate scoring, expecting 1.0; --logs captures solve.log/test-log tails below 1.0. eval_tbench2_via_api.py -- the EXACT training agent-env loop (_multi_turn) with any OpenAI-compatible API standing in for the policy: no GPU, no Ray/Megatron. For end-to-end smokes and for measuring base-model solve rates when picking a variance-band training subset. plus offline unit tests (tests/, run manually: pytest examples/experimental/openenv/tests/ -q -- not collected by the repo-level suite) for the two things a live episode cannot cheaply prove: backend dispatch on both legs, and the create-throttling retry/backoff/give-up behavior that only triggers under production rate limits. Validation: - per-task leg, no LLM (scan_golden.py): golden sweep passes 82/89 of the TB2 suite (0 infra errors; the 7 remaining have upstream-broken solutions); golden regression on this exact merge (fix-git, chess-best-move, circuit-fibsqrt all 1.0 -- covering per-task WORKDIR dispatch, apt paths, long canonical evals). - per-task leg, end-to-end with a live API policy (eval_tbench2_via_api.py): full 89-task sweep with DeepSeek as the policy (single sample per task, 30-turn cap, up to 38 concurrent episodes) -- 33/89 solved, 0 systematic infra failures (10 non-scoring episodes: 8 heavyweight-task timeouts = reward 0 under training semantics, 1 transient registry network error, 1 sandbox-resource overrun). - shared-server leg: dispatch unit test asserts behavior identical to before this change (exec prefixed with cd /app, canonical-exec scoring, rm-hack present). - unit tests: 7 passed (dispatch both legs; throttle retry/give-up/passthrough; throttle-error classification incl. the typed DaytonaRateLimitError path). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The daytona SDK is imported lazily by tbench2_env's task_snapshots (so docker-mode installs don't need it) and is not pulled in by `pip install -e tbench2_env`. On a fresh machine the per-task Daytona mode therefore fails at the WORST possible layer: every episode's sandbox start raises ModuleNotFoundError, the sample is aborted, the group is dropped by check_no_aborted, and the rollout loop refills forever -- a silent GPU-burning churn instead of an error (observed: 1900 identical failures before intervention). Fail fast instead: apply_optional_env_vars now imports daytona when OPENENV_TB2_TASKS_DIR requests the per-task backend and raises a RuntimeError with the install command at launch time. Also add the `pip install daytona` prerequisite to the README's per-task section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The per-task leg reached into the recipe module's private _make_daytona; openenv PR #966 promotes it to public API (same env-var contract: DAYTONA_API_KEY, optional DAYTONA_API_URL), so use the public name and stop depending on an underscore-private symbol that upstream review could rename without notice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

asyncio.to_thread is not cancellable: when an episode's wall-clock cap (asyncio.wait_for in run(), eval_tbench2_via_api, scan_golden) fires while create_task_sandbox is still building, the awaiting coroutine raises CancelledError but the worker thread keeps going; the finished (close_fn, url) was simply discarded, leaking a sandbox that never auto-stops (auto_stop_interval=0) until a label sweep finds it. First builds take ~10 min against 1800-3600s caps, so the window is real — the eval sweep's one "sandbox-resource overrun" is consistent with it. _create_once now records the create result thread-side; on cancellation a daemon reaper waits for the in-flight create to finish and deletes the orphan. Each retry attempt gets its own holder/event so a stale done-flag from a throttled attempt can't fool the reaper. Unit test cancels mid-create and asserts the orphan's close_fn runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The daytona-SDK preflight closed one launch-time gap but not the other half of the same failure mode: the per-task leg also hard-depends on tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which only exist on the openenv #965/#972/#966 branch. An upstream-main tbench2_env install passes the daytona check, then every episode's sandbox start fails, the sample aborts, the group drops, and the rollout loop refills forever — exactly the silent GPU-burning churn the first preflight was written to prevent. Check the recipe symbols at launch and fail with the install instruction instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The per-task backend now materializes sandboxes through the sibling tb2_task_sandbox module instead of a recipe module inside the tbench2_env package, so the launcher's recipe-symbols preflight is moot: the recipe is always present. What still varies with the install is the env SERVER the recipe bakes into each task image, so preflight that instead — probe the installed server source for the per-task contract (canonical test.sh scoring / TB2_WITHHOLD_TESTS). An unpatched install would not fail per-episode; it would silently mis-score every episode. README: pin the tbench2_env checkout the per-task leg needs (editable install — the recipe embeds the package source, which needs pyproject.toml next to the package). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The task image withholds verifier assets — solution/ never enters it — so golden replay's `cp` from /opt/tb2-tasks/<task>/solution failed the exec chain before solve.sh ever ran (solve_exit=1, no solve.log). Push the LOCAL tasks checkout's solution/ into the sandbox at golden time instead: tar.gz → base64 → chunked printf appends (64KB per exec, so the largest suite solution stages in a handful of messages) → decode at /solution. Same stage-at-use model the official harness's oracle runs use. Validated: fix-git / chess-best-move / circuit-fibsqrt golden regression 3/3 at 1.0 (solve_exit=0; circuit's 43s canonical eval intact). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The recipe module split (review feedback on #1710) moved the provider-agnostic image recipe into tb2_task_recipe; the README and the adapter/eval docstrings still said it lives in tb2_task_sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The server-side fixes this backend scores through (canonical test.sh evaluate, server-side WORKDIR, TB2_WITHHOLD_TESTS) merged upstream as huggingface/OpenEnv#965 + #972, so the README no longer pins the fork (whose pinned sha was orphaned by a rebase anyway); the launcher preflight still fails fast on an older install. scan_golden now takes the test-log tail from the evaluate output itself — the patched server wipes the on-disk copy with the verify window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

tb2_task_recipe -> tb2_sandbox_recipe, tb2_task_sandbox -> tb2_sandbox_daytona, across the adapter's import, docs, and README. Also drop the task_snapshots import alias: it dates from the pool/snapshot era and the module no longer registers snapshots on the episode path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The launcher used to forward DAYTONA_API_KEY through ray runtime_env, whose JSON is echoed by exec_command into driver logs (persisted on shared storage) and stored in ray job metadata — the key leaked in plaintext on every run. Key-supply contract now (general on purpose — files and env vars are the two forms every secret system can deliver): - workers resolve the key from their own node-local environment (DAYTONA_API_KEY: platform-injected pod env, or single-host inheritance; nothing forwards it) first, else read a key file (DAYTONA_API_KEY_FILE, default ~/.config/daytona/api_key — dotfile, K8s Secret mount, or shared-FS path); - the launcher forwards only the file PATH, never a key value, and echoes which supply is in effect; it fails fast with provisioning guidance when neither is available. Unit tests cover the resolve chain (env wins, file fallback, default path under $HOME, error when absent). Standalone tools (scan_golden, eval_tbench2_via_api) are unaffected: they resolve in-process. Note: --wandb-key rides the same echoed command line; follow-up, along with secret redaction in exec_command, tracked separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

miles core imports it (miles.rollout.generate_utils.tool_call_utils, miles.utils.debug_utils.send_to_sglang), and the openenv tbench2 adapter plus its standalone eval tool import it at module top. Installs worked only because sglang happens to depend on openai — declare it so the dependency is a contract instead of a coincidence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…nction Review follow-up on miles#1675: the generic adapter no longer knows about Daytona; mode selection is explicit plugin selection. - New openenv_daytona_agent_function.py: a drop-in --custom-agent-function-path alternative that runs every episode in its own Daytona cloud sandbox. The sandbox-create machinery (throttled retries, cancel-reap, episode env) moves there from openenv_agent_function, which keeps only the shared loop, the training wrapper, and the shared-env-server leg. - Every agent-function module exposes the same two entries: run() for miles (session-server policy wiring + training failure semantics) and run_episode() for callers that bring their own policy client and timeout/failure semantics (eval_tbench2_via_api). The per-leg differences enter the loop as three keyword parameters (run_body / native_evaluate / post_episode) filled in only by each module's run_episode — no backend objects, no env-var-sniffing dispatch. - native_evaluate replaces the old per_task flag: the branches it keys are about the SERVER CONTRACT (canonical test.sh inside `evaluate`, server-side WORKDIR — upstream since huggingface/OpenEnv#965+#972), not about per-task images; the shared docker-mode server runs per-task images too. The rm-hack becomes a post_episode hook (shared-server hygiene, unrelated to scoring). - The launcher picks the agent function from openenv_tb2_tasks_dir. Behavior change: setting OPENENV_TB2_TASKS_DIR while pointing at openenv_agent_function.run no longer switches modes implicitly. - Vocabulary: the two legs are "Daytona sandbox mode" vs "shared env server"; "per-task" is reserved for image semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…op the sample - scan_golden.py / openenv_daytona_agent_function.py: hoist the tomllib and tb2_sandbox_daytona imports to module scope (neither pulls the daytona SDK at import time). The DaytonaRateLimitError import stays in-function — its only use site, on the failure path where daytona is already in sys.modules, and older SDKs lack the class — with the comment rewritten to say so. - openenv_agent_function.py: on the native-evaluate leg, an evaluate observation with error set (or no reward at all) now yields reward=None — the server errored while scoring, which is not the same as tests failing — so the training wrapper drops the sample instead of ingesting a false-negative 0.0, matching the older leg's missing-marker semantics. Covered by a new unit test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gemini-code-assist Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread examples/experimental/openenv/openenv_launch_common.py Outdated

Comment thread examples/experimental/openenv/openenv_agent_function.py Outdated

Comment thread examples/experimental/openenv/scan_golden.py

nblintao force-pushed the tao/openenv-daytona-backend branch 4 times, most recently from 6fd9669 to 1f9ed50 Compare July 14, 2026 23:37

nblintao assigned Shi-Dong Jul 14, 2026

nblintao force-pushed the tao/openenv-daytona-backend branch from 1f9ed50 to 21d451f Compare July 14, 2026 23:48

Zhichenzzz requested a review from Shi-Dong July 14, 2026 23:49

nblintao force-pushed the tao/openenv-daytona-backend branch 3 times, most recently from 5ce2c28 to 897118c Compare July 15, 2026 01:22

Base automatically changed from shi/openenv-miles-adapter to main July 17, 2026 07:02

Shi-Dong requested review from fzyzcjy, guapisolo and yueming-yuan as code owners July 17, 2026 07:02

nblintao force-pushed the tao/openenv-daytona-backend branch from d63ca70 to 3e0b73e Compare July 17, 2026 18:30

This was referenced Jul 17, 2026

openenv/tbench2: per-task sandbox image recipe + Daytona materialization #1710

Merged

openenv/tbench2: orphan TTL + ownership labels for per-task sandboxes #1711

Merged

nblintao force-pushed the tao/openenv-daytona-backend branch from 3e0b73e to 677036c Compare July 17, 2026 21:07

nblintao changed the base branch from main to tao/tb2-sandbox-ttl July 17, 2026 21:07

nblintao force-pushed the tao/tb2-sandbox-ttl branch from 0977094 to 8b5fbb2 Compare July 17, 2026 21:37

nblintao force-pushed the tao/openenv-daytona-backend branch from 677036c to eff8e3d Compare July 17, 2026 21:37

nblintao unassigned Shi-Dong Jul 17, 2026

nblintao force-pushed the tao/openenv-daytona-backend branch from b9b951f to f48c837 Compare July 20, 2026 02:21

nblintao force-pushed the tao/tb2-sandbox-ttl branch from 8b5fbb2 to 2a23cdb Compare July 20, 2026 02:21

nblintao force-pushed the tao/openenv-daytona-backend branch from f48c837 to 231bca9 Compare July 21, 2026 23:26

nblintao force-pushed the tao/tb2-sandbox-ttl branch from 2a23cdb to fb54b3f Compare July 21, 2026 23:26

nblintao force-pushed the tao/openenv-daytona-backend branch from 231bca9 to d992bbf Compare July 21, 2026 23:28

Shi-Dong reviewed Jul 22, 2026

View reviewed changes

nblintao force-pushed the tao/tb2-sandbox-ttl branch from fb54b3f to 58a443d Compare July 22, 2026 13:44

nblintao commented Jul 23, 2026

View reviewed changes

nblintao requested a review from Shi-Dong July 23, 2026 02:09

Shi-Dong reviewed Jul 24, 2026

View reviewed changes

Shi-Dong approved these changes Jul 24, 2026

View reviewed changes

Shi-Dong reviewed Jul 24, 2026

View reviewed changes

nblintao and others added 18 commits July 24, 2026 13:52

nblintao force-pushed the tao/openenv-daytona-backend branch from a5fdd29 to 03003cc Compare July 24, 2026 17:52

nblintao merged commit 422c210 into main Jul 24, 2026
37 checks passed

nblintao deleted the tao/openenv-daytona-backend branch July 24, 2026 18:50

nblintao mentioned this pull request Jul 24, 2026

openenv/tbench2: score the shared-server leg natively; retire the adapter compensation #1790

Draft



		def _start_declarative(task_id: str, tasks_dir: str) -> tuple[Any, str]:
		import tb2_sandbox_daytona

Uh oh!

Conversation

nblintao commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Architecture (shaped by review)

Why the two legs score differently (native_evaluate)

Daytona key supply (review follow-up)

Tooling (ships with the mode; produced the validation evidence)

Validation

Post-refactor re-validation (4×H200 devbox, full stack green)

Prior evidence (larger scale, pre-split)

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nblintao commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shi-Dong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nblintao Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nblintao commented Jul 14, 2026 •

edited

Loading

Why the two legs score differently (`native_evaluate`)

nblintao Jul 24, 2026 •

edited

Loading