Skip to content

Add benchmark route profile contract#1601

Merged
huangruiteng merged 2 commits into
mainfrom
codex/benchmark-route-profile-contract-20260707
Jul 7, 2026
Merged

Add benchmark route profile contract#1601
huangruiteng merged 2 commits into
mainfrom
codex/benchmark-route-profile-contract-20260707

Conversation

@huangruiteng

Copy link
Copy Markdown
Owner

Summary

  • Add benchmark_route_profile_v0 under loopx.benchmark_core for public-safe route/model/reasoning and execution-surface metadata.
  • Encode no-local-fallback permissions for official remote benchmark routes, private auth/tunnel handle references, compact artifact policy, pacing/timeouts, failure-attribution vocabulary, and score-claim boundaries.
  • Add a focused smoke for an xhigh Codex CLI goal route without changing runner, scoring, leaderboard, or submission behavior.

Validation

  • python3 examples/benchmark-route-profile-contract-smoke.py passed.
  • python3 -m py_compile loopx/benchmark_core/route_profile.py examples/benchmark-route-profile-contract-smoke.py passed.
  • git diff --cached --check passed before commit.
  • loopx check --scan-path loopx/benchmark_core/__init__.py --scan-path loopx/benchmark_core/route_profile.py --scan-path examples/benchmark-route-profile-contract-smoke.py passed with two private-state heuristic warnings caused by intentional private/auth boundary vocabulary in the contract and smoke.
  • loopx canary premerge --from-git-diff passed direct checks, Python compile, and 6 selected catalog canaries, but did not pass the merge gate because examples/control_plane/check-public-boundary-smoke.py timed out at the canary 120s limit and benchmark-sensitive surfaces require a manual hold.
  • python3 examples/control_plane/check-public-boundary-smoke.py passed when run directly after the canary timeout.

Merge Gate

  • No benchmark jobs launched.
  • No runner/scoring/leaderboard/submission behavior changed.
  • self_merge_allowed=false from premerge canary because this is benchmark-sensitive and needs maintainer review.

@huangruiteng

Copy link
Copy Markdown
Owner Author

Findings:

  • [P1] reverse_tunnel_reference_label can still publish a local tunnel endpoint. If a caller passes a raw reverse-tunnel URL such as http://127.0.0.1:18180, build_benchmark_route_profile() sanitizes it to 127.0.0.1:18180 and validate_benchmark_route_profile() accepts it as public-safe. That preserves local endpoint/port topology in the public route profile even though the contract says transport handles are private references, not tunnel values. Please reject loopback/private host:port patterns for both transport handle labels, or coerce unsafe values to a generic handle label and add a smoke assertion for the raw URL case.

Product/architecture judgment:

The PR is useful and scoped correctly as a contract-only route profile: it does not launch runs, alter scoring, or change benchmark runner behavior. The permission policy and countability/score-claim boundaries are directionally right. The main gap is the public/private boundary for transport handle labels above.

Validation run locally on the PR worktree:

  • python3 examples/benchmark-route-profile-contract-smoke.py -> pass
  • python3 -m py_compile loopx/benchmark_core/route_profile.py loopx/benchmark_core/__init__.py examples/benchmark-route-profile-contract-smoke.py -> pass
  • git diff --check origin/main...HEAD -> pass

Merge decision: hold until the transport-handle boundary is fixed and reviewer approval is available.

@huangruiteng

Copy link
Copy Markdown
Owner Author

Validation/update after owner-authorized closeout:

  • Fixed the transport-handle blocker I raised earlier: raw endpoint/URL values such as http://127.0.0.1:18180 now collapse to a generic private handle label in build_benchmark_route_profile(), and validator rejects manually supplied endpoint labels for both auth and reverse-tunnel handles.
  • Added smoke coverage for builder fallback and validator rejection of loopback/URL transport values.

Validation run on the PR worktree:

  • python3 examples/benchmark-route-profile-contract-smoke.py -> pass
  • python3 -m py_compile loopx/benchmark_core/route_profile.py loopx/benchmark_core/__init__.py examples/benchmark-route-profile-contract-smoke.py -> pass
  • git diff --check origin/main...HEAD -> pass
  • loopx check --scan-path loopx/benchmark_core/route_profile.py --scan-path loopx/benchmark_core/__init__.py --scan-path examples/benchmark-route-profile-contract-smoke.py -> pass, public boundary scan clean
  • python3 examples/control_plane/check-public-boundary-smoke.py -> pass
  • loopx canary premerge --from-git-diff --timeout-seconds 300 -> all executed checks passed: direct diff checks, py_compile, 6 selected catalog canaries, and public-boundary smoke. It still reports manual_review_required only because benchmark-sensitive surfaces default to an explicit maintainer hold.

Merge decision: owner explicitly authorized self-merge after runnable validation in the control thread; I will treat the benchmark-sensitive manual hold as satisfied by that authorization and proceed only via admin-bypass merge, with no runner/scoring/leaderboard behavior changed.

@huangruiteng huangruiteng merged commit c5a58f1 into main Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant