Skip to content

Add opt-in targeted adaptive judging and size-aware stopping#53

Merged
davanstrien merged 2 commits into
mainfrom
feat/targeted-adaptive-stopping
Jul 15, 2026
Merged

Add opt-in targeted adaptive judging and size-aware stopping#53
davanstrien merged 2 commits into
mainfrom
feat/targeted-adaptive-stopping

Conversation

@davanstrien

@davanstrien davanstrien commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Summary

The current adaptive loop checks for convergence every five pages, but keeps judging the full all-pairs grid until every adjacent CI separates. On a 14-model run that means up to 91 pairs per page batch, so persistent near-ties can turn “adaptive” mode into almost the full benchmark.

This PR adds an opt-in targeted strategy while preserving the historical balanced default:

ocr-bench judge <output-repo> \
  --adaptive-strategy targeted \
  --size-tie-ratio 3

Targeted allocation

  • starts with enough balanced all-pairs evidence to compute an initial board
  • then tops up only adjacent pairs whose 95% CIs remain unresolved
  • recomputes adjacency after every batch, so ranking changes can activate new pairs
  • records adaptive_strategy in results metadata
  • leaves balanced as the default while the sparse strategy is validated
  • documents that conditioning later sampling on interim ranks means ordinary bootstrap CIs may be somewhat optimistic

Size-aware practical stopping

  • --size-tie-ratio RATIO treats an overlapping adjacent pair as a practical smaller-model preference when one model has at least that parameter advantage
  • applies to either adaptive strategy: it can satisfy the whole-run stopping condition in balanced, while targeted also stops topping up the pair
  • --size-tie-min-samples N requires direct pair evidence first (default: 10)
  • requires both marginal CIs to exist; missing CI/size data fails safe as unresolved
  • adds a ★ annotation to CLI, dataset rows/cards, and the viewer
  • explicitly labels this as a deployment preference: it does not change ELO/rank or claim statistical equivalence

Resume, history, and compatibility hardening

  • reconstructs the targeted pair set from checkpointed comparisons on resume instead of filling the intentionally sparse grid
  • detects new/incomplete model-grid warm-ups and continues balanced allocation until every current pair has direct evidence
  • excludes subset/retired/failed-model comparisons from the current ELO fit while preserving them in both checkpoints and final Hub publication
  • fails closed when an existing comparisons/metadata config has files but cannot be loaded, preventing transient/schema errors from becoming destructive replacement pushes
  • makes equal-ELO ordering deterministic by model name, since adjacency now controls allocation
  • stores strategy/size-rule provenance in the append-compatible metadata schema; old viewers safely ignore the new leaderboard annotation column

bench forwards the new options.

Validation

  • pytest tests/ -q594 passed
  • ruff check src/ tests/
  • ty check src/
  • independent review loop covered targeted resume, equal-ELO determinism, model-grid expansion/interruption, history preservation, fail-closed Hub reads, and missing-CI semantics

Closes #35.

@davanstrien

Copy link
Copy Markdown
Owner Author

Addressed the review in ad72d86:

  • History preservation: out-of-grid rows (subset/retired configs and currently failed models) are excluded from the current ELO fit but retained in both checkpoint payloads and final comparisons publication. Existing comparisons/metadata loads now fail closed when config files exist but cannot be loaded, so a transient/schema read failure cannot become a destructive replacement push.
  • Strategy semantics: CLI help and README now state that --size-tie-ratio applies to balanced and targeted stopping; targeted additionally stops topping up that pair.
  • Missing CIs: practical preference now requires both marginal CIs to exist.
  • Statistical caveat: README notes that targeted allocation conditions sampling on interim ranks, so ordinary bootstrap CIs may be optimistic.
  • Kept rankings_resolved as a documented public statistical-only helper and short-circuited preference recomputation when no size rule is enabled.

Added regressions for retired/subset and all-sentinel history preservation, checkpoint preservation, fail-closed Hub reads, and missing-CI behavior.

Validation: 594 tests, Ruff, ty, and CI all green.

@davanstrien
davanstrien merged commit ec802fa into main Jul 15, 2026
1 check passed
@davanstrien
davanstrien deleted the feat/targeted-adaptive-stopping branch July 15, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cost-aware adaptive stopping: stop resolving ties when one model is much smaller ("cheaper wins")

1 participant