Phase 14: Eval Optimization#184
Merged
Merged
Conversation
…ase/14-eval-optimization Why: The eval-asymmetry fix (#183) landed on develop (040239a) on top of this branch's HEAD (4c0d25c). Phase B tuning must run on the corrected eval perspective so tuning results are not contaminated by the asymmetry bug. What: Merges 44aea1a, 261e201, 040239a (eval-asymmetry correctness patch) from develop into phase/14-eval-optimization ahead of B-1/B-2/B-3 tuning runs. Left out: No new eval/search changes. Merge-only commit. Phase: 14 — Eval Optimization
engine-uci shade plugin uses shadedArtifactAttached=false (default), so the fat JAR is named engine-uci-0.5.7-SNAPSHOT.jar (no -shaded suffix). Phase: 14 — Eval Optimization
WSL2 has measured single-threaded CPU overhead vs. native Windows on the same hardware, so bench NPS numbers gathered in WSL sessions shouldn't be treated as a regression signal.
…red) Why: No GitHub MCP server exists in this environment; needed a documented, repeatable way to fetch issue content for acceptance-criteria verification. What: Added the gh CLI command to CLAUDE.md's Issue Workflow section. Left out: Did not attempt to configure an actual GitHub MCP server. Phase: 14 — eval-optimization
…ing-safety tuning Why: phase-14.md's King Safety Retune Postmortem found the 18-entry SAFETY_TABLE saturates at attacker weight >=17, giving zero Texel-tuning gradient above that point — a documented mandatory prerequisite before any further king-safety ATK-weight tuning (Issue #170, A-1). What: Extended SAFETY_TABLE to 32 entries in KingSafety.java (engine-core), TunerEvaluator.java and PositionFeatures.java (engine-tuner). Indices 0-17 are byte-for-byte unchanged (no behavior change at existing attacker-weight levels); indices 18-31 continue the table's existing diff-growth pattern (cap raised 50cp -> 160cp). Added KingSafetyTest.java asserting non-saturation at w=17..31 and correct clamping beyond the table. Documented as a dated entry in dev-entries/phase-14.md. Left out: Did not touch ATK_WEIGHT_* defaults or run any tuning — this is purely the architectural prerequisite fix. Also discarded (git checkout, not committed) an unrelated abandoned uncommitted prior tuning attempt found in the working tree (tuned_params.txt + EvalParams.java x2, ATK_WEIGHT_KNIGHT=29 vs committed baseline 6) before starting this fix. Phase: 14 — eval-optimization
…ase 15 Why: 4th attempt at king-safety ATK-weight tuning (post-SAFETY_TABLE fix) still produced worse val MSE than baseline (0.0620->0.0657), with ATK_BISHOP/ATK_QUEEN gradient-dead for all 22 iterations before early-stop. What: Documented all 4 attempts (3 prior H0 SPRTs + this run) under A-1. Root cause is corpus coverage gap for bishop/queen king-zone attack positions in tools/quiet-labeled.epd, not an optimizer or SAFETY_TABLE issue (ATK_KNIGHT/ATK_ROOK did move, confirming the table fix works). Reverted all uncommitted tuning artifacts (tuned_params.txt, both EvalParams.java) via git checkout; no king-safety values changed on this branch. SAFETY_TABLE extension (063bb1a) retained as a standalone fix. Left out: No SPRT was run — the tuning result didn't clear the bar to justify a Windows-PC SPRT run. King-safety ATK tuning deferred to Phase 15 pending corpus seed augmentation. Phase: 14 — eval-optimization
…sis falsified Why: Adam 300-iter run on mobility group (highest Fisher diagonal, no coverage issues) still produced worse val MSE than baseline (0.0620->0.0642), mirroring A-1's pattern despite mobility having no gradient-dead params. What: Tested whether Adam's fixed LR=1.0 (shared, unscaled, between full and group-restricted runs) causes early-iteration overshoot for small groups by re-running at LR=0.25. Final val MSE was virtually identical (0.064203 vs 0.064203) and the "spike" was only delayed by integer-rounding lag, not eliminated -- hypothesis falsified. Documented both attempts in dev-entries/phase-14.md. Left out: No SPRT was run; no mobility params applied to EvalParams.java. Reverted the LR=0.25 experiment (git checkout) -- GradientDescent.java LR remains at its default 1.0, no tuner hyperparameter changes committed. Phase: 14 — eval-optimization
…deferred indefinitely Why: A-3 coverage audit confirmed CONNECTED_PAWN/BACKWARD_PAWN scalars are not STARVED, ruling out corpus-coverage-gap as the explanation; single Adam pass still regressed val MSE +4.29% vs baseline, matching A-1/A-2. Task 14.7 (WDL self-play pilot) was proposed to test the distribution-mismatch hypothesis directly, but investigation found this project already has two documented self-play-WDL failure precedents (Phase 12/13): a -465 Elo material-collapse regression and a silently-broken WDL loader that produced a fake "tuned" result now sitting in tools/wdl_tuned_params.txt. What: Ran fresh --coverage-audit + single 200-iter Adam pass (K frozen at 2.773456) on pawn-structure group; documented val MSE regression and deferred per Issue #172's own acceptance-criteria escape hatch. Reverted generated artifacts (coverage-audit-report.csv, tuned_params.txt, validator-report.txt) per the established A-1 pattern for deferred runs. Investigated data/wdl-selfplay.epd (100k positions, unused since Phase 13) as a free pilot corpus for Task 14.7, confirmed it loads cleanly today, but did not use it: it predates the Phase 14 eval-asymmetry fix and SAFETY_TABLE extension, so its labels are stale relative to current eval. Left out: No self-play games generated, no Windows-PC time spent. No parameter changes applied to EvalParams.java for A-3. Task 14.7 closed without running any tuning pass against real data. Closes #172 Phase: 14 — eval-optimization
…ed baseline Why: WSL2 --bench run this session produced 77,265,370 nodes over the 31-position/depth-13 suite, vs the documented deterministic baseline of 101,771,086 nodes (bit-for-bit identical across 5 runs when it was established 2026-04-29). Search node counts are deterministic for a fixed eval+search config, so this delta means the search tree shape has changed, not that the measurement is noisy. What: Root-caused to accumulated eval changes since the baseline was set (SAFETY_TABLE 18->32, eval-asymmetry fix, PIECE_ATTACKED_BY_PAWN_MG becoming colour-relative) — all legitimately shift move ordering/pruning. Marked the 301,116/228,490 NPS gate floors suspended until a fresh native-Windows 5-run baseline exists against current HEAD. Updated #175's pre-merge checklist to reflect verified test suite results (engine-core 177/0/2, engine-tuner 131/0/1) and the still-open A-5/#174 blocker. Left out: No WSL2 re-baseline performed (per explicit instruction — WSL2 NPS is never a valid gate input). No native-Windows bench run — deferred to the same session as the #174 isolated SPRT. Phase: 14 — eval-optimization
… by #140) Why: Phase 12.2 (#130) self-play PGN->SF corpus generator. Phase 13 (#140) removed the self-play PGN extraction pipeline entirely; the current generate_texel_corpus.ps1 samples from quiet-labeled.epd instead. The debug fork (29KB) has no runtime/CI reference — only a historical dev-entry mention. Left out: everything else. Repo is disciplined; see audit report for the few remaining candidates, none safe to apply without SPRT or user sign-off.
Why: Isolated phase14-a5-contempt SPRT (871 games, TC 60+0.6, native Windows) showed contempt=50 loses -16.8 +/- 16.8 Elo vs contempt=0 (LOS 2.5%, LLR -2.97 crossed lbound -2.94) -- a clean H0, not the earlier inconclusive bundled eval-features run. Asymmetric contempt measurably hurts play at this TC. What: Reverted UciApplication.contempt default and the advertised UCI option default from 50 to 0, matching the A-1/A-2/A-3 pattern (revert value, keep infrastructure). setoption Contempt <cp> still works for opponent-specific tuning. EvalParams.CONTEMPT_THRESHOLD/CONTEMPT_VALUE scalars unchanged. SearchRegressionTest's contempt regression test calls setContempt() explicitly and is unaffected; re-verified passing. Left out: No change to EvalParams scalars or the asymmetric contemptScore() logic itself -- both stay available for future CLOP/opponent-specific tuning, just not enabled by default. Closes #174 Phase: 14 — eval-optimization
…nd gate Why: 2026-04-29 baseline (316,964 NPS / 101,771,086 nodes) was suspended 2026-07-01 after a WSL2 run showed a 24% node-count drop. Native-Windows re-run (same session as the #174 SPRT, current HEAD) confirms the drop is real -- node count matches the WSL2 run exactly (77,265,370, deterministic across all 5 runs) -- not WSL2 measurement noise. What: 5-run middle-3 protocol: mean 328,623 NPS +/-6,286 (CV 1.91%, tighter than the original 3.97%), new gate floor 312,192 NPS. This baseline supersedes 2026-04-29 for all future Phase 14/15 regression checks. Updated #175's pre-merge checklist: all A-1..A-5 verdicts recorded, both test suites verified, NPS gate satisfied by construction (this run is the new baseline). Remaining before merge: CHANGELOG.md Phase 14 entry. Left out: No merge to develop, no release.yml trigger, no pom.xml edit -- all pending explicit confirmation per CLAUDE.md and prior agreement. Phase: 14 — eval-optimization
Why: Last unchecked item on Issue #175's pre-merge checklist. Summarizes the phase's outcomes for the release notes: A-4 aspiration +156 Elo (only confirmed gain), A-5 contempt H0 with default reverted, six correctness fixes shipped independent of Elo (eval-asymmetry, backwardPawnCount, SAFETY_TABLE extension, KING_SAFETY_SCALE gradient wiring, opposite-flank shield scale, passed-pawn wiring), three deferred Texel tuning attempts (A-1/A-2/A-3, all val-MSE regressions), Task 14.7 deferred to NNUE, and the NPS baseline re-establishment. What: Inserted [0.5.7] entry above the existing [0.5.6] Phase 13 entry in CHANGELOG.md, matching its section structure (Search gains/no-change, Evaluation fixes/tuning, Tuner infra). Marked #175's dev-entries + CHANGELOG checklist item complete. Left out: No merge to develop, no release.yml trigger -- every item on #175's checklist is now satisfied, but the merge itself still needs explicit confirmation per our agreement. Phase: 14 — eval-optimization
This was referenced Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges phase/14-eval-optimization into develop per Issue #175 pre-merge checklist (all SPRT verdicts recorded, quality gates green, NPS baseline re-established on native Windows, DEV_ENTRIES.md + CHANGELOG.md complete).
Closes #175 (pending release.yml run).