From d16753620e7573b408d9b7500a6d3377bb64223f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 09:22:56 +0000 Subject: [PATCH 1/2] knowledge: Stockfish NNUE = the shipped reference for the perturbation cascade MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the operator's framing (2026-07-11): the reason to wire lance-graph + ndarray + OGAR into the stockfish work is to learn the REUSABLE Morton-cascade / inverse-pyramid perturbation-shader / HHTL / SoA compute — not a chess niche. Thesis: the workspace's perturbation-shader/HHTL surface is CONJECTURE-as-code (ndarray guid-prefix-shape-routing §3-4); Stockfish NNUE is a byte-exact PRODUCTION instance of the same structure (base + deterministic-address-indexed perturbations over a hierarchically-addressed grid, stored SoA, updated incrementally), now transcoded byte-exact (E-CHESS-TRANSCODE-COMPLETE-1). So stockfish-rs is the oracle for the reusable primitive. Contents: the NNUE-mechanism ↔ workspace-primitive correspondence table, each graded [G]/[H]/[S]; an honest mechanism-vs-rhyme ledger (deterministic-phase + stored-magnitude = [G]; Walsh-Hadamard bipolar side = NO NNUE analog, fenced); the reusable PerturbationAccumulator (ndarray mechanism / V3 policy); and a probe-gated wiring plan (D-SF-V3-1 lift-to-closure-kernel, D-PALETTE-NNUE tenant probe, D-MORTON-KA route probe, D-SF-V3-4 escalation identity). Marked SYNTHESIS — not canon until the epiphany-council + a probe run. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01Awg6TXocHcwTtc6eGsHcdD --- .../stockfish-nnue-as-perturbation-cascade.md | 151 ++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 .claude/knowledge/stockfish-nnue-as-perturbation-cascade.md diff --git a/.claude/knowledge/stockfish-nnue-as-perturbation-cascade.md b/.claude/knowledge/stockfish-nnue-as-perturbation-cascade.md new file mode 100644 index 00000000..1e5d71d2 --- /dev/null +++ b/.claude/knowledge/stockfish-nnue-as-perturbation-cascade.md @@ -0,0 +1,151 @@ +# Stockfish NNUE = the shipped, byte-exact reference for the perturbation-shader cascade + +> **READ BY:** anyone touching the perturbation shader (§4/§4b of ndarray +> `guid-prefix-shape-routing.md`), the HHTL cascade, the V3 L4 palette256² tenant, +> `PrefixShapeTable`, or the E-CHESS incrementality ruling. Companion to the +> ndarray perturbation-pyramid doc and OGAR's perturbation-encoding pins. +> +> **Status: SYNTHESIS / mostly [H], one [G] anchor.** Written to the workspace's +> own anti-theater discipline (§5 "eigenvalue theater"; the `cross-domain-synthesizer` +> mechanism-vs-rhyme test). Every correspondence is graded; the rhymes are fenced +> off from the mechanisms. NOT canon until the epiphany-council + a probe run. + +## Why this exists (the operator's framing, 2026-07-11) + +> *"The idea why lance-graph, ndarray and OGAR is to learn from Stockfish how to +> use the Morton Cascade inverse-pyramid perturbation-shader cascade — not as a +> hand-rolled niche but as reusable — and to benefit from SoA and wire Stockfish +> into V3, even if the database is fixed 90 MB; we hopefully learn how to use the +> compute 64×64 = 4096 HHTL."* + +The workspace's perturbation-shader / Morton-cascade / HHTL surface is, by its +own admission, **CONJECTURE as code** — "`PrefixShapeTable` … not yet coded" +(ndarray §3), the pyramid "CONJECTURE as code" (§4), the Walsh-Hadamard bipolar +pyramid "CONJECTURE" (§4b). We have the *algebra* (`vsa_bind`/`vsa_bundle`, +palette LUTs, HHTL escalate) and the *theory* (deterministic phase, magnitude- +only storage) but **no shipped end-to-end instance that proves the decomposition +carries a real, high-value workload byte-exactly.** + +**Stockfish NNUE is exactly that instance** — and stockfish-rs now transcodes it +byte-exact (E-CHESS-TRANSCODE-COMPLETE-1). So it is not a chess niche; it is the +**oracle for the reusable primitive**: a 30-year-hardened, production, byte-exact +implementation of *base state + deterministic-address-indexed perturbations over +a hierarchically-addressed grid, stored as an SoA, updated incrementally.* That +is the perturbation-shader cascade, shipped. + +## The correspondence — NNUE mechanism ↔ workspace primitive (graded) + +| NNUE mechanism (proven byte-exact in stockfish-rs) | Workspace primitive | Grade | Why it holds / where it strains | +|---|---|---|---| +| **Accumulator** = FT biases + Σ active-feature weight columns, per perspective, as `[[i16;1024];2] + [[i32;8];2]` | **SoA magnitude envelope** — "Lance column ≡ Arrow buffer ≡ ndarray SoA, same bytes" (§4) | **[G]** | The accumulator *is* an SoA; it is the "magnitude M — the only stored bits" of §4's decomposition. Direct, not analogy. | +| **make_index** (HalfKA: `KingBuckets[ksq]·704 + PieceSquareIndex + (s^orient^flip)`) | **HHTL cascade addressing** — coarse tier (king bucket) → fine tier (piece-square); "escalate one HHTL tier" | **[H]** | Two-level hierarchical address (bucket × square) over a 64-indexed grid = a 2-tier cascade. It is *a* hierarchy, but NOT (yet) a Morton 2bit×2bit 4×4 tile — the tiers are king-bucket/piece, not spatial quadrants. Strong structural rhyme, needs the Morton re-projection probe to become mechanism. | +| **Incremental update** (E-CHESS/L4): a move = bounded add/remove of feature columns; king move → full refresh | **The perturbation shader**: `perturb(addr,L) = M[addr@coarse] · P(phase(addr,L))`; king move = **tier escalation** (`RouteAction::Escalate`) | **[G] for the delta, [H] for the escalation identity** | The delta *is* a perturbation on a base; the king-move-refresh *is* a coarse-address change forcing recompute = exactly §5's "quorum fail → escalate one HHTL tier". The i16-wrapping group identity (`refresh(after)=refresh(before)−rm+add`) is the proven kernel (L4 self-oracle, chained). | +| **phase = make_index (which feature index is active) is DERIVED from the position/move, never stored; only the weight column is stored** | **§4 decomposition**: "phase = deterministic recurrence from the address — 0 bits stored; magnitude = the only stored bits" | **[G]** | This is the sharpest correspondence. NNUE stores weights (magnitude); *which* weights fire is computed from the board (the address). Storage scales with the weight table, not the game tree — exactly "cost scales with magnitude smoothness, not perturbation bandwidth." | +| **Bundling** = Σ feature columns, `wrapping_add` (i16) | **`vsa_bundle`** (sum + threshold), the magnitude-side algebra | **[H]** | Same *shape* (order-independent accumulation; I-SUBSTRATE-MARKOV / Chapman-Kolmogorov holds — proven order-independent in L4). But NNUE bundles **i16 magnitudes**, not ±1 bipolar signs — it is the **magnitude side only** of the two-algebra rule. There is **no sign/phase XOR side** in the accumulator. | +| **transform** (pairwise `clamp(·,0,255)²/512 → u8`) + **PSQT** | **Palette / magnitude quantizer** (`RollingFloor`); the L4 `6×palette256²` tenant | **[S]→[H]** | The transform quantizes the accumulator to u8 for the affine — analogous to palette quantization, but it is a fixed nonlinearity, not a 256-centroid codebook lookup. Rhyme unless re-expressed as a palette LUT (a probe). | +| **int8 affine layers** (`fc_0/1/2`) via `matmul_i8_to_i32` | **ndarray `simd_runtime::matmul_i8_to_i32`** (the shared compute) | **[G]** | Literally the same function. Already wired (stockfish-rs L5). | +| **bucket = (pieces−1)/4 → LayerStack[bucket]** | **classid-prefix dispatch → ClassView** | **[H]** | A prefix (piece count) selecting a compute lens = a classid selecting a ClassView. Structural; the "8 buckets" ↔ "8 render lenses" is a rhyme until a ClassView actually dispatches the stack. | +| **64 × 64 = 4096** (king-square × piece-square-index; and the butterfly from-to lane) | **the 64×64 = 4096 gridlake / 1BRC lane J sweet spot** | **[G] on the number, [H] on the identity** | The 4096 is real and load-bearing in both. Whether NNUE's 64×64 is *the same* 4096 as the gridlake (vs. a coincident cardinality) is the open question the wiring answers. | + +## Mechanism vs rhyme — the honest ledger (do not skip) + +**Genuine mechanism (transfer these):** +- **Deterministic-address phase + stored-magnitude** ([G]). NNUE *proves* you can + store only the envelope and regenerate "which cells fire" from the address. This + de-risks PROBE-PERT-RHO: a shipped workload already lives at "magnitude-only". +- **Incremental perturbation with coarse-tier escalation** ([G]/[H]). NNUE proves + the base+delta update is byte-exact and that a coarse-address change (king move) + cleanly forces a tier refresh — the exact shape of `RouteAction::Escalate`. +- **SoA + int8 GEMM** ([G]). Already shared code. + +**Rhyme (do NOT ship as if proven — fence them):** +- **Walsh-Hadamard / bipolar sign pyramid (§4b) has NO analog in the NNUE + accumulator.** NNUE is magnitude-only bundling; there is no ±1 sign/phase side, + no XOR algebra, no superposition-unbind. Claiming NNUE "is" the Walsh-Hadamard + cascade would be eigenvalue theater. NNUE informs the **magnitude side**; §4b's + sign side is a *separate* conjecture NNUE does not witness. +- **Morton 2bit×2bit 4×4 tile** is not NNUE's addressing — NNUE's hierarchy is + king-bucket × piece, not spatial-quadrant Morton. The "cascade" is real; the + "Morton" is aspirational until the re-projection probe (below). +- **palette256² L4 tenant** is not how NNUE stores weights (i16, not palette + pairs). The similarity-as-one-table-read is a rhyme unless a probe shows NNUE's + distance structure survives palette quantization at the ρ anchors. + +## The reusable primitive (the thing that must not stay chess-niche) + +**A `PerturbationAccumulator` — an SoA over a HHTL-addressed feature grid, updated +by bounded, deterministic-address-indexed perturbations, with coarse-tier +escalation.** Domain-agnostic. Chess is one consumer; the compute is the point. + +- **ndarray = MECHANISM** (per `data-flow.md` + §2 "ndarray is mechanism, never + policy"): the SoA accumulator kernel — `refresh` / `apply_delta` (i-wrapping + group op) / `escalate` — parameterized by *(feature-count, lane-width, an + address→active-index closure, a stored magnitude table)*. This is stockfish-rs's + `Accumulator` + `HalfKaAccumulator` **generalized to drop the chess types**: the + `make_index` closure and the weight table become inputs. That is the whole + reusability move — lift L3/L4 from `Chess`-typed to closure-parameterized, land + the kernel in ndarray, and stockfish-rs becomes a *consumer* that supplies the + chess closure + the 90 MB table. +- **lance-graph / V3 = POLICY**: the accumulator's SoA *is* a V3 tenant lane; the + address is a `NodeGuid` prefix; the perturbation is the shader over that lane; + the escalation is `RouteAction::Escalate` on the HHTL cache. + +## Wiring Stockfish into V3 (the plan — probe-gated, not asserted) + +Stated as deliverables so it can be executed and falsified, never hand-waved: + +- **D-SF-V3-1 — lift the accumulator to a closure-parameterized ndarray kernel.** + Generalize `Accumulator::{refresh, apply-delta}` to + `PerturbationAccumulator` taking an + `active_indices: impl Fn(&Addr) -> SmallVec` + a `&MagnitudeTable`. + stockfish-rs re-expresses its L3/L4 as the chess instantiation. **Gate:** the + existing L3/L4 byte-exact oracles STILL pass through the generic kernel (no + regression — the reference stays green). +- **D-SF-V3-2 — the NNUE accumulator AS a V3 SoA tenant.** Map `[[i16;1024];2]` + onto a V3 tenant lane (le-contract): is it an L4 palette pair carrier, or a new + raw-i16 lane? **Probe D-PALETTE-NNUE:** does palette256²-quantizing the FT + weight columns preserve the eval within the ρ anchor (Pflug-10 / Jirak floor)? + If YES → NNUE weights are a genuine L4 tenant (huge: 90 MB → palette-compressed + + one-table-read similarity). If NO → NNUE needs a raw-magnitude lane and the + palette-tenant rhyme is fenced. *This is the single highest-value probe — it + tells us whether the frozen 90 MB net is a palette tenant or not.* +- **D-SF-V3-3 — make_index → HHTL/Morton route.** Probe D-MORTON-KA: re-project + HalfKA's king-bucket × piece-square addressing onto a Morton 2bit×2bit tile and + measure whether nearest-in-Morton ⇒ nearest-in-feature (the quorum τ). Confirms + or kills the "NNUE 4096 = gridlake 4096" identity ([H] → [G] or dropped). +- **D-SF-V3-4 — the escalation identity.** Assert (test) that NNUE's + king-move-refresh == `RouteAction::Escalate` at a named tier — the first place + the chess reference and the HHTL cache share a code path. + +## What the frozen 90 MB net buys us (the operator's point, made precise) + +The net is fixed — that is a **feature for this purpose**: a frozen, byte-exact, +externally-validated magnitude table lets us test the *compute* (addressing, +perturbation, escalation, palette-quantization, one-table-read similarity) +against a ground truth that cannot drift. We are not learning chess; we are using +chess as the **only workload in the building where "did the perturbation cascade +reproduce the answer byte-for-byte?" has a yes/no oracle.** Every probe above is +gradeable precisely because the net is frozen. + +## Fences (no theater) + +1. This doc is SYNTHESIS. The [G] anchors are the accumulator-as-SoA, the + deterministic-phase/stored-magnitude decomposition, and the shared int8 GEMM. + Everything else is [H]/[S] and **named as such** — do not cite the rhymes as + proven. +2. NNUE witnesses the **magnitude side** of the two-algebra rule only. It says + nothing about the bipolar Walsh-Hadamard sign side (§4b) — keep them separate. +3. No deliverable here ships without its probe green (the workspace's own §5 + quorum-or-escalate discipline turned on our own claims). + +## Cross-refs + +- ndarray `guid-prefix-shape-routing.md` §4 (perturbation pyramid), §4b + (Walsh-Hadamard), §5 (φ-quorum / anti-theater). +- V3 `soa_layout/le-contract.md` (L1–L4 tenant ladder; L4 = 6×palette256²). +- OGAR `CLAUDE.md` — "Perturbation encoding — DETERMINISTIC PHASE", + "Bipolar-phase pyramid", "256×256 CENTROID TILE", "256 = 4⁴ hierarchy". +- Board: `E-CHESS-TRANSCODE-COMPLETE-1`, `E-CHESS` #539. +- `AdaWorldAPI/stockfish-rs`: L3 `src/eval/accumulator.rs`, L4 + `src/eval/incremental.rs` (the reference kernel to lift), L5 `network_eval.rs` + (the int8 GEMM consumer). From 8db5e6eb50a2cd937b1cc5118e7a6f0cc0dba9dd Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 09:30:09 +0000 Subject: [PATCH 2/2] =?UTF-8?q?knowledge:=20NNUE=20temporal/episodic=20lay?= =?UTF-8?q?er=20=E2=80=94=20game=20=3D=20Markov=20version-stream?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extends the perturbation-cascade synthesis with the temporal axis: a chess game is a temporal.rs version-stream, each apply_move is one Markov step AND the spatial L4 delta (one accumulator, two axes). Opening variants = L1 part_of:is_a basins; transpositions = L2 memberof:members; episodic recall = AriGraph EpisodicMemory. SurrealQL AST fenced as query adapter only (not the episodic spine, per SURREAL-AST-TRAP). Four probe-gated deliverables (D-SF-EPISODIC-1/-BASIN-1/-RUNG-1/-ARIGRAPH-1); the strongest reuses the existing incremental oracle as a temporal-replay oracle at zero new ground-truth cost. Graded [G]/[H]/[S]; SYNTHESIS-not-canon. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01Awg6TXocHcwTtc6eGsHcdD --- .../stockfish-nnue-as-perturbation-cascade.md | 87 ++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-) diff --git a/.claude/knowledge/stockfish-nnue-as-perturbation-cascade.md b/.claude/knowledge/stockfish-nnue-as-perturbation-cascade.md index 1e5d71d2..5c644845 100644 --- a/.claude/knowledge/stockfish-nnue-as-perturbation-cascade.md +++ b/.claude/knowledge/stockfish-nnue-as-perturbation-cascade.md @@ -138,11 +138,96 @@ gradeable precisely because the net is frozen. 3. No deliverable here ships without its probe green (the workspace's own §5 quorum-or-escalate discipline turned on our own claims). +## The temporal/episodic layer — a chess game IS a Markov version-stream + +The operator's extension: *"we could wire the AriGraph episodic and SurrealQL +AST time series and/or episodic basins = opening variants."* The chess workload +witnesses the **temporal** substrate as cleanly as it witnesses the spatial one — +and the two are the SAME operation seen from two axes. + +**The key unification.** The NNUE incremental update +(`incremental.rs::apply_move`) is BOTH: +- a **spatial** perturbation — the L4 delta over the 64×64 board grid + (add/remove piece features, king-refresh on escalation); AND +- one **temporal** Markov step — ply *n* → ply *n+1* along the game stream. + +There is no separate "temporal engine." A game is the accumulator's own trajectory +through version-space, and each `apply_move` is exactly one `temporal.rs` stream +entry. The spatial cascade and the temporal stream are one accumulator, read on +two axes. + +| Chess object | Temporal substrate | Grade | +|---|---|---| +| A game (ply sequence) | a `temporal.rs` sorted version-stream; ply *v* = one Lance version | **[G]** — moves already carry a total order; `apply_move` IS the step | +| "position after ply *v*" | `QueryReference::at(v, rung)` + deinterlace — a zero-copy projection, no replay | **[H]** — the read is [G]; that it needs no recomputation is the D-SF-EPISODIC-1 gate | +| Analysis horizon (present vs hindsight) | the **rung**: low rung = reason strictly at ply *v*; high rung = spoiler-read the game's outcome to label the move (training-target labeling) | **[H]** — rung semantics are shipped; the chess *use* (hindsight = eval-vs-result delta) is the D-SF-RUNG-1 probe | +| Opening variants (e4/d4/…) | **episodic basins** = le-contract **L1 `part_of:is_a`** rails (a variant is-a line is-a opening) | **[H]** — rails are [G]; that opening-trees fit the L1 rail shape is the D-SF-BASIN-1 probe | +| Transpositions (same position, different move-order) | **L2 `memberof:members`** — one position node, member-of many variant basins | **[H]** — the many-basins-one-node shape is exactly L2; the probe confirms it round-trips | +| Episodic recall ("games that reached this pawn structure") | AriGraph `EpisodicMemory` / `markov_soa` — retrieve-similar over position fingerprints | **[G]** for the store; **[H]** that chess positions are good episodic keys (D-SF-ARIGRAPH-1) | +| Time-series query over games | SurrealQL AST — a **query adapter** over the version-stream | **[G]** — adapter only; see the fence | + +### Fence — SurrealQL AST is the query adapter, NOT the episodic spine + +The episodic logic (basins, transpositions, rung-selection, the Markov step) lives +in the accumulator trajectory + the le-contract rails + AriGraph. **SurrealQL AST +is only the read/time-series *adapter* over that stream** — per the OGAR +SURREAL-AST-TRAP: behavior never lives in DDL, and a `DEFINE EVENT … WHEN … THEN` +carrying game-lifecycle logic is the negative-beauty hijack the doctrine rejects. +Games flow: `apply_move` → temporal-stream version → le-contract basin → +(optionally) a SurrealQL SELECT projects it. Never: SurrealQL DDL *drives* the game. + +### Probe-gated deliverables (temporal) + +- **D-SF-EPISODIC-1 — a game as a temporal version-stream.** Model an N-ply game + as N `temporal.rs` entries; read position-at-ply-*v* via `QueryReference::at`. + **Gate:** the projected accumulator at ply *v* == the accumulator freshly + computed from the ply-*v* FEN, byte-for-byte — *reusing the L4 chained oracle + already green in `incremental.rs`*. This is the **strongest temporal probe**: + it turns the existing incremental oracle into a temporal-replay oracle at zero + new ground-truth cost. If it passes, "position-at-version is a zero-copy + projection" is [H]→[G]. +- **D-SF-BASIN-1 — opening variants as L1 basins.** Encode a small opening tree + (e.g. 3 openings × 2 variants) as L1 `part_of:is_a` rails; assert a position + resolves to its basin set. **Gate:** transposition (same FEN via two move-orders) + lands on ONE position node that is `memberof` both variant basins (L2) — the + many-basins-one-node round-trip. +- **D-SF-RUNG-1 — hindsight labeling via rung.** For a decided game, read each + move at a **low rung** (eval as-of that ply) and at a **high rung** (spoiler-read + the result); the low↔high delta = the "was this move objectively good given the + outcome?" signal. **Gate:** the two reads differ exactly where eval and result + disagree (blunders that won, brilliancies that lost) — proving rungs carry the + present/hindsight horizon, not just a version index. +- **D-SF-ARIGRAPH-1 — positions as episodic keys.** Store game positions in + AriGraph `EpisodicMemory`; `retrieve_similar(position_fp)` returns games that + reached a near structure. **Gate:** retrieval precision above the Jirak noise + floor on a labeled set (e.g. all games with an isolated queen's-pawn) — MEASURED, + never asserted. + +### Why this closes the loop + +Spatial (the board), temporal (the game), and episodic (the archive of games) are +**three reads of one accumulator trajectory** — the same `apply_move` delta, +projected on the square axis (perturbation cascade), the version axis (Markov +stream), and the similarity axis (episodic recall). The frozen 90 MB net keeps all +three honest: every temporal projection has the same byte-exact yes/no oracle the +spatial cascade has, because replay-to-ply-*v* and compute-from-ply-*v*-FEN must +agree to the bit. That is the whole reason chess is the learning vehicle — it is +the only workload in the building where *all three axes* answer to one frozen +ground truth. + ## Cross-refs - ndarray `guid-prefix-shape-routing.md` §4 (perturbation pyramid), §4b (Walsh-Hadamard), §5 (φ-quorum / anti-theater). -- V3 `soa_layout/le-contract.md` (L1–L4 tenant ladder; L4 = 6×palette256²). +- V3 `soa_layout/le-contract.md` (L1–L4 tenant ladder; L1 `part_of:is_a` + + L2 `memberof:members` = episodic basins/transpositions; L4 = 6×palette256²). +- `temporal.rs` — `QueryReference::at(v, rung)` + deinterlace (the game + version-stream); `E-MARKOV-TEMPORAL-STREAM-1` (Markov moves off the VSA braid + onto the sorted stream — the ruling the temporal layer here rides on). +- AriGraph — `EpisodicMemory` / `markov_soa` / `triplet_graph` (the episodic + store; positions as episodic keys). +- OGAR `SURREAL-AST-TRAP-PREFLIGHT.md` — SurrealQL AST is a query adapter, not + the episodic spine (behavior never in DDL). - OGAR `CLAUDE.md` — "Perturbation encoding — DETERMINISTIC PHASE", "Bipolar-phase pyramid", "256×256 CENTROID TILE", "256 = 4⁴ hierarchy". - Board: `E-CHESS-TRANSCODE-COMPLETE-1`, `E-CHESS` #539.