Add benchmark route profile contract#1601
Conversation
|
Findings:
Product/architecture judgment: The PR is useful and scoped correctly as a contract-only route profile: it does not launch runs, alter scoring, or change benchmark runner behavior. The permission policy and countability/score-claim boundaries are directionally right. The main gap is the public/private boundary for transport handle labels above. Validation run locally on the PR worktree:
Merge decision: hold until the transport-handle boundary is fixed and reviewer approval is available. |
|
Validation/update after owner-authorized closeout:
Validation run on the PR worktree:
Merge decision: owner explicitly authorized self-merge after runnable validation in the control thread; I will treat the benchmark-sensitive manual hold as satisfied by that authorization and proceed only via admin-bypass merge, with no runner/scoring/leaderboard behavior changed. |
Summary
benchmark_route_profile_v0underloopx.benchmark_corefor public-safe route/model/reasoning and execution-surface metadata.Validation
python3 examples/benchmark-route-profile-contract-smoke.pypassed.python3 -m py_compile loopx/benchmark_core/route_profile.py examples/benchmark-route-profile-contract-smoke.pypassed.git diff --cached --checkpassed before commit.loopx check --scan-path loopx/benchmark_core/__init__.py --scan-path loopx/benchmark_core/route_profile.py --scan-path examples/benchmark-route-profile-contract-smoke.pypassed with two private-state heuristic warnings caused by intentional private/auth boundary vocabulary in the contract and smoke.loopx canary premerge --from-git-diffpassed direct checks, Python compile, and 6 selected catalog canaries, but did not pass the merge gate becauseexamples/control_plane/check-public-boundary-smoke.pytimed out at the canary 120s limit and benchmark-sensitive surfaces require a manual hold.python3 examples/control_plane/check-public-boundary-smoke.pypassed when run directly after the canary timeout.Merge Gate
self_merge_allowed=falsefrom premerge canary because this is benchmark-sensitive and needs maintainer review.