diff --git a/evals/docs/litert-faithfulness-bypass.md b/evals/docs/litert-faithfulness-bypass.md new file mode 100644 index 0000000..e63e876 --- /dev/null +++ b/evals/docs/litert-faithfulness-bypass.md @@ -0,0 +1,131 @@ +# LiteRT-LM reason-faithfulness bypass + +Measures whether the on-device LiteRT-LM model (Qwen3-0.6B-int4) stays faithful +to the deterministic tags the Move Coach prompt supplies — the measurement the +roadmap's Sequence 2 calls for, done without building a LiteRT-LM provider into +ferryman (ferryman is HTTP-only; the generator lives here). + +## Why a bypass + +ferryman's harness has the reason-faithfulness scorer. The LiteRT-LM generator +lives in this app. Wiring LiteRT-LM into ferryman as a provider would be a +multi-day feature (native deps, model payload, the MCP tool-call loop — which a +0.6B model can't reliably do anyway). The scorer is provider-agnostic, so we +decouple: **this app generates, ferryman scores.** One JSON contract between them. + +## Two-step run + +### 1. Generate (this repo — produces `litert-outputs.json`) + +```bash +cd /path/to/compose-multiplatform-chess +git checkout eval/litert-faithfulness-bypass # or main, once merged + +# Drive LiteRT-LM over 10 golden cases, emit outputs JSON. +# First run downloads the ~347 MB Qwen3-0.6B-int4 model (~1-2s to load after). +# The driver lives in :litert-eval — its own module, isolated from :server so +# its dependency graph stays Ktor-free (see build.gradle.kts for why). +# litert-outputs.json lands in litert-eval/. Use an absolute path to write elsewhere: +# --args="10 /tmp/litert-outputs.json" +./gradlew :litert-eval:run --args="10 litert-outputs.json" +``` + +Output shape (one object per case): + +```json +{ + "id": "opening-001", + "fen": "rnbqkbnr/...", + "bestMoveUci": "g1h3", + "tags": ["opening", "develops"], + "output": "Nh3 develops the knight toward the center.", + "route": "litert", + "firstTokenMs": 1200, + "completeMs": 3400 +} +``` + +`route` is `"litert"` for a real generation, or `"fallback ()"` when the +model produced nothing and the driver recorded the deterministic `MoveCoachFallback` +text (what the production orchestrator would show the user). + +### 2. Score (ferryman repo — produces the scorecard) + +```bash +cd /path/to/ferryman +python3 eval_harness/score_litert_outputs.py /path/to/litert-outputs.json \ + --write scorecard-litert.md +``` + +The scorer imports ferryman's `_check_faithfulness` and scores each output +against the pre-supplied tags (no FEN derivation — candidates.json carries +hand-authored tags per case). Two axes per case: + +- **Faithfulness** — paraphrase covers the supplied tag concepts AND asserts no + unsupported high-stakes concept (check / checkmate / wins material / capture / + castle / promotion). The "Nf3 attacks the enemy king" failure mode (tag says + `develops`, model invents a threat) fails here. +- **Honesty** — the forbidden-phrase gate (no "Stockfish thinks" / "engine depth" + / unsupported certainty). Same gate the chess app's + `MoveCoachResponseValidator` enforces client-side. + +No API keys, no network — pure local scoring. + +## What the scorecard tells you + +The roadmap's prediction to confirm: **the honesty gate passes at high rates** +(the model is paraphrasing supplied tags, not inventing engine evals), and the +question is **whether faithfulness holds** — does the paraphrase cover the tags, +or does the 0.6B model add unsupported detail (wrong piece, invented threat)? + +A high honesty / low faithfulness split is the most informative outcome: it +means the existing client-side validator (honesty + grounding) is *insufficient*, +and the faithfulness scorer catches what it can't — which is the gap the roadmap +names. That's the finding worth reporting. + +## Why a separate `:litert-eval` module + +The driver lives in its own Gradle module (`:litert-eval`), not in `:evals`, +because of an internally-inconsistent dependency: **litertlm-jvm 0.14.0's +bytecode and its POM disagree about kotlinx-coroutines.** + +- litertlm-jvm's `sendMessageAsync` bytecode calls + `SendChannel.close$default(SendChannel, Throwable, int, Object)` — a static + bridge that **only exists in kotlinx-coroutines 1.11.0+**. +- litertlm-jvm's POM declares kotlinx-coroutines-core-jvm **1.9.0** (which lacks + the bridge). So the version the POM asks for cannot satisfy the bytecode. +- The rest of this repo resolves coroutines to **1.10.2** (Ktor 3.4.3, via + `:server`) or **1.9.0** (the catalog) — neither provides the method. + +The result: litertlm crashes with `NoSuchMethodError: SendChannel.close$default` +the moment a real generation completes (compile is fine; the call site is in a +JNI callback that only runs at inference time). + +The fix is module-scoped: `:litert-eval` depends on `:onDeviceAi` + `:coachapi` +only (no `:server`, no Ktor) and forces coroutines to **1.11.0** via +`resolutionStrategy.force` in `litert-eval/build.gradle.kts`. That's the single +version that provides the bridge litertlm needs. + +If you ever move the driver back into `:evals`, or add a `:server`/Ktor +dependency here, Ktor will drag coroutines back to 1.10.2 and the crash returns. +Don't. When litertlm-jvm publishes a build whose bytecode matches its POM (or +Ktor moves to 1.11.0+), this isolation can be reconsidered — but verify by +running the driver to completion (not just compiling) before merging modules. + +## Fixture note + +The driver runs against `evals/golden/candidates.json`, whose tag vocabulary was +normalized to the `MoveCoachFallback` tag constants (`develops`, `center-control`, +`king-safety`, `opening`; `pawn-tension` has no canonical constant and passes +through `describeTags`' `else -> tag` as a literal string). Before this +normalization, the tags used `development` / `center`, which didn't match +`describeTags`' `when` branches — the prompt was handing the model raw strings +instead of the translated phrases. Fixing the fixture was the article's thesis +applied to itself: a fixture bug silently misleads every measurement. + +## Hardware requirements + +- Apple Silicon Mac (M-series) or Linux x86_64 — `litertlm-jvm` ships native + libs for `darwin-aarch64`, `linux-x86_64`, `linux-aarch64`, `win-x86_64`. +- **Intel Mac is unsupported** (no `darwin-x86_64` native lib). The driver + detects this and exits with a clear message rather than emitting empty outputs. diff --git a/evals/golden/candidates.json b/evals/golden/candidates.json index 51b73f2..9d3b109 100644 --- a/evals/golden/candidates.json +++ b/evals/golden/candidates.json @@ -5,7 +5,7 @@ "bestMoveUci": "g1h3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -22,7 +22,7 @@ "bestMoveUci": "f2f4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -70,7 +70,7 @@ "bestMoveUci": "g8f6", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -94,7 +94,7 @@ "bestMoveUci": "a2a3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -111,7 +111,7 @@ "bestMoveUci": "b2b4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -130,7 +130,7 @@ "bestMoveUci": "f2f3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -147,7 +147,7 @@ "bestMoveUci": "d8h4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -167,7 +167,7 @@ "bestMoveUci": "c2c3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -190,7 +190,7 @@ "bestMoveUci": "b1c3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -211,7 +211,7 @@ "bestMoveUci": "e1f2", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -230,7 +230,7 @@ "bestMoveUci": "h2h3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -247,7 +247,7 @@ "bestMoveUci": "g2g4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -266,7 +266,7 @@ "bestMoveUci": "e7e5", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -344,7 +344,7 @@ "bestMoveUci": "a2a3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -363,7 +363,7 @@ "bestMoveUci": "g2g4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -380,7 +380,7 @@ "bestMoveUci": "f7f5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -398,7 +398,7 @@ "bestMoveUci": "g7g5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -416,7 +416,7 @@ "bestMoveUci": "f2f4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -435,7 +435,7 @@ "bestMoveUci": "f1g2", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -454,7 +454,7 @@ "bestMoveUci": "c7c6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -495,7 +495,7 @@ "bestMoveUci": "c2c4", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -542,7 +542,7 @@ "bestMoveUci": "c2c3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -565,7 +565,7 @@ "bestMoveUci": "b2b3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -588,7 +588,7 @@ "bestMoveUci": "c7c6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -610,7 +610,7 @@ "bestMoveUci": "g8e7", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -638,7 +638,7 @@ "bestMoveUci": "b8c6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -658,7 +658,7 @@ "bestMoveUci": "d5d4", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -680,7 +680,7 @@ "bestMoveUci": "g4g5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -701,7 +701,7 @@ "bestMoveUci": "c2c4", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -722,7 +722,7 @@ "bestMoveUci": "b1c3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -743,7 +743,7 @@ "bestMoveUci": "h7h5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -765,7 +765,7 @@ "bestMoveUci": "d2d3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -788,7 +788,7 @@ "bestMoveUci": "g2g3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -805,7 +805,7 @@ "bestMoveUci": "f7f5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -831,7 +831,7 @@ "bestMoveUci": "d2d3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -856,7 +856,7 @@ "bestMoveUci": "b2b4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -877,7 +877,7 @@ "bestMoveUci": "e7e6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -897,7 +897,7 @@ "bestMoveUci": "f7f5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -915,7 +915,7 @@ "bestMoveUci": "g8f6", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -933,7 +933,7 @@ "bestMoveUci": "g7g5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -951,7 +951,7 @@ "bestMoveUci": "g7g6", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -998,7 +998,7 @@ "bestMoveUci": "g1f3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1017,7 +1017,7 @@ "bestMoveUci": "b2b4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1040,7 +1040,7 @@ "bestMoveUci": "c7c5", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1060,7 +1060,7 @@ "bestMoveUci": "f3h4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1081,7 +1081,7 @@ "bestMoveUci": "c7c5", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1099,7 +1099,7 @@ "bestMoveUci": "c7c6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1119,7 +1119,7 @@ "bestMoveUci": "g7g6", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1137,7 +1137,7 @@ "bestMoveUci": "h5h4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1157,7 +1157,7 @@ "bestMoveUci": "b2b3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1180,7 +1180,7 @@ "bestMoveUci": "h2h4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1197,7 +1197,7 @@ "bestMoveUci": "h1h3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1216,7 +1216,7 @@ "bestMoveUci": "a7a5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1234,7 +1234,7 @@ "bestMoveUci": "b2b4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1253,7 +1253,7 @@ "bestMoveUci": "c2c3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1276,7 +1276,7 @@ "bestMoveUci": "c2c3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1297,7 +1297,7 @@ "bestMoveUci": "e2e4", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1318,7 +1318,7 @@ "bestMoveUci": "g7g5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1336,7 +1336,7 @@ "bestMoveUci": "d2d3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1357,7 +1357,7 @@ "bestMoveUci": "h7h5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1375,7 +1375,7 @@ "bestMoveUci": "d2d3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1392,7 +1392,7 @@ "bestMoveUci": "g2g4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1411,7 +1411,7 @@ "bestMoveUci": "e7e5", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1429,7 +1429,7 @@ "bestMoveUci": "g2g3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1450,7 +1450,7 @@ "bestMoveUci": "b2b4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1467,7 +1467,7 @@ "bestMoveUci": "d7d5", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1485,7 +1485,7 @@ "bestMoveUci": "g1f3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1506,7 +1506,7 @@ "bestMoveUci": "c8f5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1526,7 +1526,7 @@ "bestMoveUci": "c7c5", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1544,7 +1544,7 @@ "bestMoveUci": "b4b5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1565,7 +1565,7 @@ "bestMoveUci": "a2a3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1584,7 +1584,7 @@ "bestMoveUci": "d7d6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1604,7 +1604,7 @@ "bestMoveUci": "f7f5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1622,7 +1622,7 @@ "bestMoveUci": "d8d6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1642,7 +1642,7 @@ "bestMoveUci": "b8c6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1660,7 +1660,7 @@ "bestMoveUci": "g8h6", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1678,7 +1678,7 @@ "bestMoveUci": "g7g6", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1698,7 +1698,7 @@ "bestMoveUci": "e2e4", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1746,7 +1746,7 @@ "bestMoveUci": "a2a4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1767,7 +1767,7 @@ "bestMoveUci": "c7c6", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1785,7 +1785,7 @@ "bestMoveUci": "b4b5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1806,7 +1806,7 @@ "bestMoveUci": "b7b6", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1828,7 +1828,7 @@ "bestMoveUci": "a7a6", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1877,7 +1877,7 @@ "bestMoveUci": "e2e3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1900,7 +1900,7 @@ "bestMoveUci": "e2e4", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1923,7 +1923,7 @@ "bestMoveUci": "b7b5", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -1941,7 +1941,7 @@ "bestMoveUci": "e2e4", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1962,7 +1962,7 @@ "bestMoveUci": "c7c5", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1982,7 +1982,7 @@ "bestMoveUci": "c2c3", "tags": [ "opening", - "center" + "center-control" ], "eco": "A00", "movesSan": [ @@ -1999,7 +1999,7 @@ "bestMoveUci": "b1a3", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -2016,7 +2016,7 @@ "bestMoveUci": "a1b1", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -2041,7 +2041,7 @@ "bestMoveUci": "g2g4", "tags": [ "opening", - "development" + "develops" ], "eco": "A00", "movesSan": [ @@ -2055,4 +2055,3 @@ ] } ] - diff --git a/litert-eval/.gitignore b/litert-eval/.gitignore new file mode 100644 index 0000000..eca4a35 --- /dev/null +++ b/litert-eval/.gitignore @@ -0,0 +1,2 @@ +# Generated by the :litert-eval:run driver — scored by ferryman, not committed. +litert-outputs.json diff --git a/litert-eval/build.gradle.kts b/litert-eval/build.gradle.kts new file mode 100644 index 0000000..23005f9 --- /dev/null +++ b/litert-eval/build.gradle.kts @@ -0,0 +1,37 @@ +plugins { + alias(libs.plugins.kotlinJvm) + alias(libs.plugins.kotlinSerialization) + application +} + +dependencies { + // The driver needs :onDeviceAi for LitertLmTextGenerator / + // MoveCoachPromptBuilder and :coachapi for the request types. It does NOT + // depend on :server — keeping Ktor out of this module's graph means the + // coroutines version here is controlled solely by the force below, not + // dragged to 1.10.2 by Ktor 3.4.3. (See the force block for why 1.11.0.) + implementation(project(":ondeviceai")) + implementation(project(":coachapi")) + implementation(libs.kotlinx.coroutines.core) + implementation(libs.kotlinx.serialization.json) +} + +kotlin { + jvmToolchain(21) +} + +application { + mainClass.set("com.example.literteval.EvalLiteRtDriverKt") +} + +// litertlm-jvm 0.14.0 is an internally-inconsistent artifact: its bytecode +// calls `SendChannel.close$default(SendChannel, Throwable, int, Object)` — a +// static bridge that ONLY exists in kotlinx-coroutines 1.11.0+ — but its POM +// declares coroutines 1.9.0 (which lacks the bridge). So the version the POM +// asks for cannot satisfy the bytecode. 1.11.0 is the minimum that provides +// the method; we force it here. (1.10.2, the default resolution elsewhere in +// this repo via Ktor 3.4.3, also lacks the bridge — that's the original crash.) +configurations.all { + resolutionStrategy.force("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.11.0") + resolutionStrategy.force("org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.11.0") +} diff --git a/litert-eval/src/main/kotlin/com/example/literteval/EvalLiteRtDriver.kt b/litert-eval/src/main/kotlin/com/example/literteval/EvalLiteRtDriver.kt new file mode 100644 index 0000000..3b13f2d --- /dev/null +++ b/litert-eval/src/main/kotlin/com/example/literteval/EvalLiteRtDriver.kt @@ -0,0 +1,210 @@ +package com.example.literteval + +import com.example.ondeviceai.AiAvailability +import com.example.ondeviceai.AiTokenOrFinal +import com.example.ondeviceai.MoveCoachFallback +import com.example.ondeviceai.MoveCoachPromptBuilder +import com.example.ondeviceai.MoveCoachRequest +import com.example.ondeviceai.litertlm.LitertLmModelStore +import com.example.ondeviceai.litertlm.LitertLmTextGenerator +import kotlinx.coroutines.flow.toList +import kotlinx.coroutines.runBlocking +import kotlinx.serialization.Serializable +import kotlinx.serialization.builtins.ListSerializer +import kotlinx.serialization.json.Json +import java.nio.file.Files +import java.nio.file.Path +import kotlin.system.exitProcess + +/** + * Standalone driver that runs the LiteRT-LM on-device generator over the golden + * set and emits raw outputs as JSON, for scoring by ferryman's + * `score_reason_faithfulness` scorer. + * + * Lives in its own Gradle module (`:litert-eval`) deliberately: this module + * depends on `:onDeviceAi` + `:coachapi` only — **not** `:server` — so its + * dependency graph stays free of Ktor, and the kotlinx-coroutines version here + * is controlled solely by the `resolutionStrategy.force` in + * `litert-eval/build.gradle.kts`. That force is the actual reason this module + * exists: litertlm-jvm 0.14.0 is an internally-inconsistent artifact whose + * bytecode calls `SendChannel.close$default` (a static bridge that only exists + * in coroutines 1.11.0+) while its POM declares 1.9.0. The rest of the app + * resolves to 1.10.2 (via Ktor 3.4.3) or 1.9.0 (catalog) — neither provides + * the method, so litertlm crashes the moment a generation completes. Forcing + * 1.11.0 here is the only version that satisfies litertlm's bytecode. + * + * Why a separate driver (not a route in EvalMain): [EvalMain] drives + * `FakeTextGenerator` + the opening-explainer HTTP routes, scored by the chess + * app's own EvalScorer (grounding + length only — the weak check). This + * driver's purpose is different: produce the 0.6B model's *actual* paraphrases + * under the Move Coach prompt contract, so an external scorer (ferryman's + * reason-faithfulness check) can measure whether those paraphrases stay faithful + * to the supplied deterministic tags. Generation and scoring are deliberately + * decoupled — two repos, two languages, one JSON contract between them. + * + * Usage: + * ./gradlew :litert-eval:run --args="[caseCount] [outputPath]" + * # defaults: 10 cases, writes ./litert-outputs.json + * + * The driver downloads the ~347MB Qwen3-0.6B-int4 model on first run + * ([LitertLmModelStore]); subsequent runs load from disk (~1-2s). On unsupported + * hosts (Intel Mac — no darwin-x86_64 native lib) it exits with a clear message + * rather than emitting empty outputs. + * + * Run from the repository root so `golden/candidates.json` resolves (the + * `application` plugin's `run` task sets the working dir to this module's dir, + * so the default path below points one level up). + */ +fun main(args: Array) { + val caseCount = args.getOrNull(0)?.toIntOrNull() ?: 10 + val outputPath = Path.of(args.getOrNull(1) ?: "litert-outputs.json") + + // Load cases directly from the golden set (mirrors GoldenCase's shape) — + // depending on :evals would pull Ktor via :server, defeating the isolation. + val cases = loadCandidates(Path.of("../evals/golden/candidates.json")).take(caseCount) + println("LiteRT-LM faithfulness driver: ${cases.size} cases → $outputPath") + + // Download the model if missing (blocking, with progress to stdout). + if (!LitertLmModelStore.isDownloaded()) { + println("Downloading LiteRT-LM model (~347 MB, first run only)...") + LitertLmModelStore.download { frac -> + print("\r ${((frac * 100).toInt()).coerceIn(0, 100)}%") + } + println() + } + + val generator = LitertLmTextGenerator( + modelPath = LitertLmModelStore.modelFile().absolutePath, + ) + + runBlocking { + // Warm up the engine once so the first case isn't penalized for model load. + val status = generator.status() + if (status is AiAvailability.Error) { + System.err.println( + "LiteRT-LM unavailable: ${status.message}\n" + + "(Intel Mac is unsupported — litertlm-jvm ships no darwin-x86_64 native lib.)", + ) + exitProcess(1) + } + generator.warmup() + println("Engine ready. Running ${cases.size} cases...") + + val records = cases.mapIndexed { index, case -> + val record = runOneCase(generator, case) + print(" [${index + 1}/${cases.size}] ${case.id}: ${record.route}") + record.firstTokenMs?.let { print(" firstToken=${it}ms") } + println() + record + } + + val json = Json { prettyPrint = true } + Files.writeString(outputPath, json.encodeToString(ListSerializer(OutputRecord.serializer()), records)) + println("Wrote ${records.size} outputs to $outputPath") + println("Next: score them with ferryman's reason-faithfulness scorer:") + println(" python3 score_litert_outputs.py $outputPath") + } + runBlocking { generator.close() } +} + +private suspend fun runOneCase( + generator: LitertLmTextGenerator, + case: CandidateCase, +): OutputRecord { + val request = case.toMoveCoachRequest() + val prompt = MoveCoachPromptBuilder.build(request) + + val chunks = try { + generator.generate(prompt).toList() + } catch (t: Throwable) { + // An inference failure becomes a fallback record — the scorer will see + // the deterministic fallback text and score it (which should pass + // faithfulness, since the fallback is built from the same tags). + return fallbackRecord(case, reason = "generation failed: ${t.message}") + } + + val text = tokenText(chunks) + val metrics = chunks.lastOrNull { it is AiTokenOrFinal.Final } as? AiTokenOrFinal.Final + + return if (text.isNotBlank()) { + OutputRecord( + id = case.id, + fen = case.fen, + bestMoveUci = case.bestMoveUci, + tags = case.tags, + output = text, + route = "litert", + firstTokenMs = metrics?.metrics?.firstTokenMs, + completeMs = metrics?.metrics?.completeMs, + ) + } else { + // Empty generation → the orchestrator would fall back in production. + fallbackRecord(case, reason = metrics?.metrics?.fallbackReason ?: "empty output") + } +} + +private fun fallbackRecord(case: CandidateCase, reason: String): OutputRecord { + // The production orchestrator falls back to MoveCoachFallback when the + // generator produces nothing. Record that text so the scorer sees what + // the user would actually see, and mark the route. + val fallbackText = MoveCoachFallback.build(case.toMoveCoachRequest()) + return OutputRecord( + id = case.id, + fen = case.fen, + bestMoveUci = case.bestMoveUci, + tags = case.tags, + output = fallbackText, + route = "fallback ($reason)", + firstTokenMs = null, + completeMs = null, + ) +} + +private fun tokenText(chunks: List): String = buildString { + chunks.forEach { chunk -> + when (chunk) { + is AiTokenOrFinal.Token -> append(chunk.text) + is AiTokenOrFinal.Final -> append(chunk.text) + } + } +}.trim() + +/** Mirrors com.example.evals.GoldenCase — read directly to avoid the :evals dep. */ +@Serializable +private data class CandidateCase( + val id: String, + val fen: String, + val bestMoveUci: String, + val tags: List, + val eco: String? = null, + val movesSan: List = emptyList(), + val expectedConcepts: List = emptyList(), +) + +private fun loadCandidates(path: Path): List { + val json = Json { ignoreUnknownKeys = true } + return json.decodeFromString(ListSerializer(CandidateCase.serializer()), Files.readString(path)) +} + +private fun CandidateCase.toMoveCoachRequest() = MoveCoachRequest( + fenBefore = fen, + bestMoveUci = bestMoveUci, + bestMoveDisplay = bestMoveUci, + sideToMove = if (" b " in fen) "Black" else "White", + evaluationBeforeCp = null, + evaluationAfterCp = null, + deterministicTags = tags, + engineDifficultyName = "EVAL", +) + +@Serializable +data class OutputRecord( + val id: String, + val fen: String, + val bestMoveUci: String, + val tags: List, + val output: String, + val route: String, + val firstTokenMs: Long?, + val completeMs: Long?, +) diff --git a/onDeviceAi/src/desktopMain/kotlin/com/example/ondeviceai/litertlm/LitertLmTextGenerator.kt b/onDeviceAi/src/desktopMain/kotlin/com/example/ondeviceai/litertlm/LitertLmTextGenerator.kt index f4ec436..1026818 100644 --- a/onDeviceAi/src/desktopMain/kotlin/com/example/ondeviceai/litertlm/LitertLmTextGenerator.kt +++ b/onDeviceAi/src/desktopMain/kotlin/com/example/ondeviceai/litertlm/LitertLmTextGenerator.kt @@ -158,7 +158,20 @@ class LitertLmTextGenerator( ) companion object { - const val DEFAULT_CONTEXT_SIZE = 2048 + /** + * Must be <= the model's KV-cache capacity. The Qwen3-0.6B-int4 model + * (`qwen3_0.6b_q4_block32_ekv1280.litertlm`) exposes `max_num_tokens: 1280` + * in its metadata — the `ekv1280` in the filename is the extended KV-cache + * size. Requesting more (the old default of 2048) compiles fine but the + * native runtime traps with SIGTRAP (JVM exit 133) once a generation's + * prompt + `` chain-of-thought + output crosses 1280 tokens — + * typically on checkmate/mate-in-N cases that trigger long reasoning. + * The trap is a C++ CHECK inside litertlm-jvm, so it can't be caught by + * Kotlin's `try/catch (Throwable)` and kills the whole driver. 1024 + * leaves headroom under the 1280 cap while still fitting the Move Coach + * prompt + a full paraphrase. + */ + const val DEFAULT_CONTEXT_SIZE = 1024 const val DEFAULT_TOP_K = 40 const val DEFAULT_TOP_P = 1.0 } diff --git a/settings.gradle.kts b/settings.gradle.kts index 2ac9dcd..2dcc7ef 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -83,4 +83,5 @@ include(":coachapi") project(":coachapi").projectDir = file("coachApi") include(":server") include(":evals") +include(":litert-eval") include(":perft-mcp")