You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Enable agentic RL through black-box agent harnesses (Pi, OpenCode, Claude Code, …) inside OpenEnv. The harness issues LLM calls at an OpenAI-compatible boundary; an interception server fronts a trainer-controlled engine, captures token-faithful traces of those calls, and feeds them to TRL — with rewards graded inside SWE environment. This is the installed-agent training path (train through the harness as-is), the counterpart to the external-agent pattern TRL's Harbor integration already covers.
The end state — one env, one agent, plus the core infra they need:
swe_rl_env — the single SWE environment for this work (SWE-rebench V2 task substrate, sandboxed test execution, binary all-pass grading). Works black-box on its own.
swe_rl_agent — the agent harness swe_rl_env drives (Pi, OpenCode, Claude Code, … via the RFC 005 CLIHarnessAdapter seam). Black-box: only its base URL + API key are injected.
Sandbox backends in core — SandboxBackend/SandboxHandle/BgJob with E2B, Docker, and HF backends, reusable across envs.
Interception + token-faithful trace (OpenEnv core) — an OpenEnv-hosted interception server (session-ID-as-API-key auth) fronting TRL's vLLM, and a trace recorder emitting prompt_ids, response_ids, loss_mask, response_logprobs, reward with prefix-break forking. TRL keeps generation (return_token_ids), advantages, weight sync, and IS correction; the seam is rollout_func via CLIHarnessAdapter.run_white_box.
Ownership split with TRL: OpenEnv owns the env, sandbox backends, interception server, and token-faithful trace recorder; TRL owns generation (vLLM return_token_ids), advantages/GRPO, weight sync, IS correction, and chat-template correctness; the rollout_func seam is co-designed. The RFC and swe_rl_env (black-box) start now with no dependency; the interception + trace PRs land against TRL's stabilized rollout_func/vllm-serve seam and the RFC 005 core/harness/ runtime.
Goal
Enable agentic RL through black-box agent harnesses (Pi, OpenCode, Claude Code, …) inside OpenEnv. The harness issues LLM calls at an OpenAI-compatible boundary; an interception server fronts a trainer-controlled engine, captures token-faithful traces of those calls, and feeds them to TRL — with rewards graded inside SWE environment. This is the installed-agent training path (train through the harness as-is), the counterpart to the external-agent pattern TRL's Harbor integration already covers.
The end state — one env, one agent, plus the core infra they need:
swe_rl_env— the single SWE environment for this work (SWE-rebench V2 task substrate, sandboxed test execution, binary all-pass grading). Works black-box on its own.swe_rl_agent— the agent harnessswe_rl_envdrives (Pi, OpenCode, Claude Code, … via the RFC 005CLIHarnessAdapterseam). Black-box: only its base URL + API key are injected.SandboxBackend/SandboxHandle/BgJobwith E2B, Docker, and HF backends, reusable across envs.prompt_ids, response_ids, loss_mask, response_logprobs, rewardwith prefix-break forking. TRL keeps generation (return_token_ids), advantages, weight sync, and IS correction; the seam isrollout_funcviaCLIHarnessAdapter.run_white_box.run_white_boxseam (doesn't parallel it); records draft [Draft] core: harness-rollout contract + runnable example for online RL with TRL #864 as superseded prior art.Ownership split with TRL: OpenEnv owns the env, sandbox backends, interception server, and token-faithful trace recorder; TRL owns generation (vLLM
return_token_ids), advantages/GRPO, weight sync, IS correction, and chat-template correctness; therollout_funcseam is co-designed. The RFC andswe_rl_env(black-box) start now with no dependency; the interception + trace PRs land against TRL's stabilizedrollout_func/vllm-serveseam and the RFC 005core/harness/runtime.CC: @burtenshaw