Skip to content

feat: Switchyard OpenCode integration #2080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
drifold wants to merge 8 commits into
base: feature/switchyard-openhands-integration
Choose a base branch
from
Open

feat: Switchyard OpenCode integration #2080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5708ff7
feat: opencode upstream support through Switchyard token capture
drifold Jul 19, 2026
2b3a34d
fix: canonicalize tool-call args for switchyard multi-turn history co…
drifold Jul 19, 2026
20125ee
fix(swe_agents): steer upstream opencode to omit invalid task_id via …
drifold Jul 20, 2026
6d868e1
feat: per-run switchyard instance spawning for token injection
drifold Jul 21, 2026
a01d40e
feat(swe_agents): default opencode harness to upstream, split fork co…
drifold Jul 23, 2026
b2008cb
feat(swe_agents): derive opencode model limits from Switchyard's serv…
drifold Jul 23, 2026
32f991a
fix(swe_agents): round-robin Switchyard spawns across generation back…
drifold Jul 29, 2026
8afdd31
fix(swe_agents): correct Switchyard spawn for full-scale training
drifold Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,8 +317,10 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
| Swe Agents | | - | - | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_multi_tools.yaml'>swebench_multi_tools.yaml</a> | - |
| Swe Agents | | - | - | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_openhands.yaml'>swebench_openhands.yaml</a> | - |
| Swe Agents | | - | - | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_openhands_training.yaml'>swebench_openhands_training.yaml</a> | - |
| Swe Agents | coding | SWE-bench driven by the opencode agent framework for RL training. | Train software engineering capabilities on SWE tasks using opencode rollouts. | ✓ | - | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_opencode_training.yaml'>swebench_opencode_training.yaml</a> | - |
| Swe Agents | coding | SWE-bench driven by the opencode agent framework. | Eval software engineering capabilities on SWE-bench using opencode. | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_opencode.yaml'>swebench_opencode.yaml</a> | - |
| Swe Agents | coding | SWE-bench driven by the opencode agent framework for RL training. | Train software engineering capabilities on SWE tasks using opencode rollouts. | ✓ | - | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_nv_opencode_training.yaml'>swebench_nv_opencode_training.yaml</a> | - |
| Swe Agents | coding | SWE-bench driven by the opencode agent framework. | Eval software engineering capabilities on SWE-bench using opencode. | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_nv_opencode.yaml'>swebench_nv_opencode.yaml</a> | - |
| Swe Agents | coding | SWE-bench driven by upstream opencode for RL training. | Train software engineering capabilities on SWE tasks using upstream opencode rollouts. | ✓ | - | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_opencode_training.yaml'>swebench_opencode_training.yaml</a> | - |
| Swe Agents | coding | SWE-bench driven by upstream opencode. | Eval software engineering capabilities on SWE-bench using upstream opencode. | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_opencode.yaml'>swebench_opencode.yaml</a> | - |
| Swe Pivot | agent | SWE pivot verifier for PivotRL on coding agent trajectories | Improve coding agent fix-design decisions | ✓ | ✓ | Apache 2.0 | <a href='resources_servers/swe_pivot/configs/swe_pivot.yaml'>swe_pivot.yaml</a> | - |
| Swerl Gen | coding | Running sandboxed evaluation for SWE-style tasks (either patch generation or reproduction test generation) | Improve SWE capabilities useful for benchmarks like SWE-bench | ✓ | ✓ | Apache 2.0 | <a href='resources_servers/swerl_gen/configs/swerl_gen.yaml'>swerl_gen.yaml</a> | - |
| Swerl Llm Judge | coding | SWE-style multiple-choice LLM-judge tasks scored via <solution>...</solution> choice. | Improve SWE capabilities useful for benchmarks like SWE-bench | ✓ | ✓ | MIT | <a href='resources_servers/swerl_llm_judge/configs/swerl_llm_judge.yaml'>swerl_llm_judge.yaml</a> | - |
Expand Down
39 changes: 35 additions & 4 deletions nemo_gym/switchyard_trace.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
masks the sample rather than emitting partially annotated training data.
"""

import json
import math
from typing import Any, Dict, List, NamedTuple

Expand Down Expand Up @@ -106,9 +107,7 @@ def reconstruct_switchyard_rollout(envelope: Any, session_id: str, converter: Re
for i, record in enumerate(records):
_validate_record(record, i, session_id, record_uuids)
if record["prompt_token_ids"][: len(token_history)] != token_history:
raise SwitchyardTraceError(
f"record {i} prompt_token_ids do not extend the prior prompt+generation tokens"
)
raise SwitchyardTraceError(f"record {i} prompt_token_ids do not extend the prior prompt+generation tokens")
token_history = record["prompt_token_ids"] + record["generation_token_ids"]
if record["model"] != model:
raise SwitchyardTraceError(f"record {i} model {record['model']!r} != session model {model!r}")
Expand All @@ -123,7 +122,11 @@ def reconstruct_switchyard_rollout(envelope: Any, session_id: str, converter: Re
if i == 0:
new_context = prompt
else:
if prompt[: len(assembled)] != assembled:
# Compare with tool-call arguments canonicalized so a client that
# re-serializes JSON differently across turns (compact vs. pretty)
# does not fail the extension check. The reconstructed items keep the
# original arguments — this normalization is comparison-only.
if _canonicalize_tool_args(prompt[: len(assembled)]) != _canonicalize_tool_args(assembled):
raise SwitchyardTraceError(f"record {i} prompt does not extend the reconstructed history")
new_context = prompt[len(assembled) :]
if not new_context:
Expand Down Expand Up @@ -255,6 +258,34 @@ def _content_to_text(content: Any, i: int) -> str:
raise SwitchyardTraceError(f"record {i} contains unsupported content of type {type(content).__name__}")


def _canonicalize_tool_args(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Copy of *messages* with assistant tool-call ``arguments`` canonicalized to
sorted-key JSON, used only for the history-extension equality check.

A client may re-serialize the same tool-call JSON differently across turns
(compact vs. pretty-printed); that whitespace difference must not fail the
extension check. The reconstructed training items keep the original arguments
— this normalization never reaches them.
"""
out: List[Dict[str, Any]] = []
for message in messages:
tool_calls = message.get("tool_calls")
if not tool_calls:
out.append(message)
continue
canonical_calls = []
for call in tool_calls:
function = call["function"]
arguments = function["arguments"]
try:
arguments = json.dumps(json.loads(arguments), sort_keys=True)
except (json.JSONDecodeError, TypeError):
pass # not valid JSON — compare as-is
canonical_calls.append({**call, "function": {**function, "arguments": arguments}})
out.append({**message, "tool_calls": canonical_calls})
return out


def _normalize_tool_calls(tool_calls: Any, i: int) -> List[Dict[str, Any]]:
if not isinstance(tool_calls, list):
raise SwitchyardTraceError(f"record {i} contains non-list tool_calls")
Expand Down
6 changes: 4 additions & 2 deletions responses_api_agents/swe_agents/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,8 +409,10 @@ Bundled YAML configs:
- `configs/swebench_openhands_training.yaml` — same shape as above but tuned for training.
- `configs/swe_agent_config.yaml` — alternative SWE-agent path (uses `agent_tools_file`).
- `configs/swebench_multi_tools.yaml` — full 15-way prompt × agent-class × tool-name bundle (OpenHands).
- `configs/swebench_opencode.yaml` — opencode harness (`agent_framework: opencode`), pinned fork + commit, no prompt overrides.
- `configs/swebench_opencode_training.yaml` — opencode harness tuned for training.
- `configs/swebench_opencode.yaml` — upstream opencode harness (`opencode_source: opencode`), routed through Switchyard, no prompt overrides.
- `configs/swebench_opencode_training.yaml` — upstream opencode harness tuned for training.
- `configs/swebench_nv_opencode.yaml` — NeMo-Gym opencode fork (`opencode_source: nv-opencode`), pinned repo + commit. Deprecated; prefer the upstream configs above.
- `configs/swebench_nv_opencode_training.yaml` — fork harness tuned for training. Deprecated; prefer the upstream config above.
- `configs/swebench_opencode_multi_tools.yaml` — opencode harness with the 11-way `prompts/opencode_harness/*.txt` prompt bundle. See [Prompt overrides](#opencode-integration).
- `configs/swebench_opencode_no_instruction.yaml` / `configs/swebench_opencode_empty.yaml` — opencode ablation baselines (minimal / no methodology instructions).
- `configs/swebench_deepswe.yaml` / `configs/swebench_denovoswe.yaml` — the `deepswe` / `denovoswe` datasets (⚠️ WIP, see [Supported datasets and harnesses](#supported-datasets-and-harnesses)).
Expand Down
Loading