Skip to content

Move trusted tool preparation into arena#191

Open
vaibhav-i wants to merge 1 commit into
codex/arena-separation-trusted-script-quarantinefrom
codex/arena-separation-trusted-tool-preparation
Open

Move trusted tool preparation into arena#191
vaibhav-i wants to merge 1 commit into
codex/arena-separation-trusted-script-quarantinefrom
codex/arena-separation-trusted-tool-preparation

Conversation

@vaibhav-i

Copy link
Copy Markdown
Collaborator

Summary

  • move pinned Bubblewrap, Codex, Claude Code, and optional Turso installation policy into ax-arena/benchmark/scripts/prepare-trusted-tools.sh
  • leave GitHub-only inputs, secret binding, permissions, environment review, and artifact actions in the workflow launcher
  • preserve the existing credential-free preparation job and strengthen shell failure behavior

Validation

  • npm test — 559 core tests and 132 arena tests passed on Node 22
  • npm run typecheck
  • npm run build
  • npm run pack:check
  • both npm dry-run packs
  • npm run check:boundaries
  • bash -n ax-arena/benchmark/scripts/prepare-trusted-tools.sh
  • git diff --check
  • independent security and behavior-parity audit: clean

Compatibility / rollback

Workflow inputs, tool pins, install destinations, smoke tests, credentials, and artifact flow are unchanged. Rollback is a single commit revert.

Next slice

Move arena-only environment template ownership out of the public ax-eval package without changing active runtime aliases.

@github-actions

Copy link
Copy Markdown

Normalized records diff

Overall is a macro-average of participating surface scores. Agents are never averaged together.

Overall by agent

Product Agent Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Surfaces
fixture-db claude-code 92.9% 92.9% +0.0 pp 78.6% (11/14) 78.6% (11/14) 2
fixture-db codex 78.6% 78.6% +0.0 pp 64.3% (9/14) 64.3% (9/14) 2

Surface cells

Product Agent Surface Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Latency ms Cost USD Tokens Harness version
fixture-db claude-code api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1100 → 1100 0.4200 → 0.4200 3350 → 3350 2.1.5 → 2.1.5
fixture-db claude-code cli 100.0% 100.0% +0.0 pp 85.7% (6/7) 85.7% (6/7) 1300 → 1300 0.5100 → 0.5100 3720 → 3720 2.1.5 → 2.1.5
fixture-db codex api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1200 → 1200 — → — 3600 → 3600 0.121.0 → 0.121.0
fixture-db codex cli 71.4% 71.4% +0.0 pp 57.1% (4/7) 57.1% (4/7) 1500 → 1500 — → — 4100 → 4100 0.121.0 → 0.121.0

Operational metrics are context only and never affect ranking.

@vaibhav-i
vaibhav-i force-pushed the codex/arena-separation-trusted-script-quarantine branch from e91445e to f1169ba Compare July 22, 2026 15:00
@vaibhav-i
vaibhav-i force-pushed the codex/arena-separation-trusted-tool-preparation branch from f816bc4 to aeab6bf Compare July 22, 2026 15:00
@vaibhav-i
vaibhav-i force-pushed the codex/arena-separation-trusted-script-quarantine branch from f1169ba to 3ee1aa8 Compare July 22, 2026 21:21
@vaibhav-i
vaibhav-i force-pushed the codex/arena-separation-trusted-tool-preparation branch from aeab6bf to cf485ed Compare July 22, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant