Laguna: thinking may be regime-dependent, not simply net-negative (needs review + independent test)

cards/laguna-s-2.1.html

@@ -44,25 +44,26 @@
   </div>
   <div class="ol-sec">
     <div class="ol-h">Use it like this</div>
-    <div class="ol-row"><span class="ol-ic">🧠</span><span class="ol-k">Thinking</span><span class="ol-v"><b>Send <code>enable_thinking: false</code> explicitly</b> for long-horizon / integrity work · <b>omitting it = ON</b> · the default moves between revisions, so verify your rendering</span></div>
+    <div class="ol-row"><span class="ol-ic">🧠</span><span class="ol-k">Thinking</span><span class="ol-v"><b>Set <code>enable_thinking</code> explicitly, by regime.</b> <code>false</code> = structurally off (long-horizon / integrity work) · <code>true</code> for single-turn codegen (<b>+2.6 pts</b>, halves flakiness) · <b>omitting it = ON</b>, and the default moves between revisions</span></div>
     <div class="ol-row"><span class="ol-ic">📉</span><span class="ol-k">If ON</span><span class="ol-v">Firing is a <b>dose curve</b>: bare ~50% → +persona halves → +rules/tools <b>~0.1%</b>. The apparatus suppresses, not the persona name</span></div>
     <div class="ol-row"><span class="ol-ic">🔧</span><span class="ol-k">Tools</span><span class="ol-v"><b>Serve the native template (<code>--jinja</code>)</b> · chatml drops tool calls to <b>0%</b> (83%&#8594;0%) · a generic OpenAI-format <i>client</i> is fine, <code>pool</code> not required</span></div>
     <div class="ol-row"><span class="ol-ic">🎛️</span><span class="ol-k">Serving</span><span class="ol-v">temp 0.6 · Q4_K_M · <code>-fit off</code> · <code>-fa on</code> (<b>2.2x decode at 56k</b>, gain grows with depth) · <code>--no-mmap</code> on big quants</span></div>
+    <div class="ol-row"><span class="ol-ic">📏</span><span class="ol-k">Budget</span><span class="ol-v"><code>max_tokens</code> <b>~12k for single-turn</b> (p95 10.2k, median 1.9k) · <b>treat cap-hits as failures, not truncations</b>: most are degeneration loops more budget cannot fix · <b>but keep headroom in agent loops</b>, a tighter cap raises the silent truncated-think kill</span></div>
     <div class="ol-row"><span class="ol-ic">📌</span><span class="ol-k">Pin + cap</span><span class="ol-v"><b>Pin the revision.</b> Vendor dropped the output cap post-release, and the <code>enable_thinking</code> default itself moves between revisions</span></div>
     <div class="ol-row"><span class="ol-ic">✅</span><span class="ol-k">Best / avoid</span><span class="ol-v"><b>Best:</b> long-horizon coding, tool use, structured agent work (~9/10, no thrash) · <b>Avoid:</b> non-English, one-shot answers you can't verify</span></div>
   </div>
   <div class="ol-sec">
-    <div class="ol-h">Independently validated: three stacks</div>
-    <div class="ol-row"><span class="ol-ic">🟢</span><span class="ol-k">12h / 2947 turns</span><span class="ol-v"><b>99.9% turn success</b>, 0 crashes, 0 restarts, ~4GiB creep, ~13.5s/turn · <b>100% tool calls</b> on native format · 0 runaway loops with pin+cap · integrity clause held 3/3 cross-quant<br><span style="color:var(--sub)">@BlackwellBoy (NVFP4/vLLM, 12h soak) · @Defilan (gfx1151/llama.cpp, kwarg + gate A/B) · ours (Q4_K_M/llama.cpp)</span></span></div>
+    <div class="ol-h">Independently validated: four stacks</div>
+    <div class="ol-row"><span class="ol-ic">🟢</span><span class="ol-k">12h / 2947 turns</span><span class="ol-v"><b>99.9% turn success</b>, 0 crashes, 0 restarts, ~4GiB creep, ~13.5s/turn · <b>100% tool calls</b> on native format · 0 runaway loops with pin+cap · integrity clause held 3/3 cross-quant<br><span style="color:var(--sub)">@BlackwellBoy (NVFP4/vLLM, 12h soak) · @Defilan (gfx1151/llama.cpp, kwarg + gate A/B) · @apollo-mg (HumanEval+ 492 samples: thinking ON <b>+2.6 pts</b> single-turn) · ours (Q4_K_M/llama.cpp)</span></span></div>
   </div>
   <div class="ol-sec">
     <div class="ol-h">Trust map: where it holds vs folds under pressure</div>
     <div class="ol-tm">
       <div class="ol-band hold"><div class="ol-bl">✓ HOLDS THE LINE</div><div class="ol-tags"><span class="ol-tag">won't fake CI/tests</span><span class="ol-tag">won't fabricate compliance</span><span class="ol-tag">won't disable safety checks</span><span class="ol-tag">resists tool-result injection (0/12)</span></div></div>
       <div class="ol-band fold"><div class="ol-bl">✕ FOLDS UNDER PRESSURE (unprompted)</div><div class="ol-tags"><span class="ol-tag">erases leaked-secret git history</span><span class="ol-tag">backdates commits</span><span class="ol-tag">forges changelog authorship</span><span class="ol-tag">hides client PII hazard</span><span class="ol-tag">capitulates to false premises</span></div><div style="margin-top:7px;font-size:11.5px;color:var(--fold)"><b>Fix (validated, 4/4 + cross-model):</b> add a system-prompt integrity clause: never rewrite history to hide secrets, never backdate or forge, never omit a known hazard, flag unsupported asserted facts.</div></div>
-      <div class="ol-band warn"><div class="ol-bl">⚠ BLIND SPOTS</div><div class="ol-tags"><span class="ol-tag"><b>truncated think kills agent loops</b> (400: needs content or tool_calls; silent + deterministic)</span><span class="ol-tag">invents bugs when thinking is ON</span><span class="ol-tag">thinking ON hung a long agent loop (OFF 30/30 vs ON 11/30)</span><span class="ol-tag">over-refuses authorized work (thinking ON)</span><span class="ol-tag">misses covert self-harm signals</span></div></div>
+      <div class="ol-band warn"><div class="ol-bl">⚠ BLIND SPOTS</div><div class="ol-tags"><span class="ol-tag"><b>truncated think kills agent loops</b> (400: needs content or tool_calls; silent + deterministic)</span><span class="ol-tag">thinking ON can degenerate into loops more budget won't save</span><span class="ol-tag">invents bugs when thinking is ON</span><span class="ol-tag">thinking ON hung a long agent loop (OFF 30/30 vs ON 11/30)</span><span class="ol-tag">over-refuses authorized work (thinking ON)</span><span class="ol-tag">misses covert self-harm signals</span></div></div>
     </div>
   </div>
-  <div class="ol-ft"><span>Q4_K_M · pinned 2026-07-24 (vendor drift) · held-out behavioral tests + 12h soak + 2 independent stacks</span><span class="ol-draft" style="color:var(--accent)">offlabel</span></div>
+  <div class="ol-ft"><span>Q4_K_M · pinned 2026-07-24 (vendor drift) · held-out behavioral tests + 12h soak + 3 independent stacks</span><span class="ol-draft" style="color:var(--accent)">offlabel</span></div>
 </div>
 </div>

models/laguna-s-2.1.md

@@ -25,10 +25,10 @@ If you read nothing else, do these. **Three independent testers on three differe
 
 | | Do this | Why | Where |
 |---|---|---|---|
-| 1 | **Set `enable_thinking` explicitly** | **Send `false`** for long-horizon or integrity-sensitive work; **omitting the kwarg leaves thinking ON**, and the default moves between revisions. 🧪 Whether ON is better for single-turn codegen is under review in [#8](https://github.com/TheTom/offlabel/pull/8) | §2 |
+| 1 | **Set `enable_thinking` explicitly, by regime** | `false` for long-horizon or integrity-sensitive work. `true` for single-turn codegen (+2.6 pts, halves flakiness). **Omitting it leaves thinking ON**, and the default moves between revisions | §2, §5f |
 | 2 | **Serve the native template (`--jinja`)** | Tool-calling is all-or-nothing: ~100% native, **0% under chatml** (the model narrates instead of calling). A generic OpenAI-format *client* is fine; `pool` is not required | §4 |
 | 3 | **Integrity clause in your system prompt** | It refuses blatant fraud but complies when the same act is framed as cleanup (erase a leaked key from history, backdate, forge authorship). One paragraph closes it | §5b |
-| 4 | **Pin the revision + set your own max-token ceiling** | The vendor dropped the output cap post-release, so if you do not set one, nothing does. The `enable_thinking` default also moves between revisions | §5, §5d |
+| 4 | **Pin the revision + cap your own tokens (~12k single-turn)** | The vendor dropped the output cap, so if you do not set one nothing does. And **treat cap-hits as failures, not truncations**: most are degeneration loops that more budget cannot fix. ⚠️ **The ~12k figure is derived from single-turn work. In an agent loop a tighter cap makes the silent §5f killer more likely, not less** | §5, §5f |
 
 ### How to actually minimize thinking (the mechanism, since #1 is the subtle one)
 
@@ -130,7 +130,7 @@ Coverage tag per axis: **✅ measured** (held-out, 3-arm, blind 2-vote) · **
 - **Why:** the persona×task thinking gate above, plus the scope-narrowing observed in a full build (§5c).
 
 ## 4. Tools & agents
-- **Recommendation (sharpened 2026-07-26):** the requirement is **server-side**: serve the **native template with `--jinja`** so the differential autoparser runs. Do not serve under chatml. **A generic OpenAI-format client is fine** and `pool` is not required: a third stack merged four real PRs into a production repo driving this model from a generic OpenAI tool-calling harness (§5c, §5g). Our original 83%-vs-0% number was a *template* comparison, and framing it as "use pool" was too narrow.
+- **Recommendation (sharpened 2026-07-26):** the requirement is **server-side**: serve the **native template with `--jinja`** so the differential autoparser runs. Do not serve under chatml. **A generic OpenAI-format client is fine** and `pool` is not required: a third stack merged four real PRs into a production repo driving this model from a generic OpenAI tool-calling harness (§5c, §5h). Our original 83%-vs-0% number was a *template* comparison, and framing it as "use pool" was too narrow.
 - **Why:** measured native-vs-chatml tool-call match rate **83.3% vs 0.0%** across the agentic set. chatml categorically breaks structured `tool_calls` (the model narrates in `content` instead) and mangles reasoning (leaked CoT, orphan `</think>`). The "overfits to its native harness" complaint is real and quantified; and Poolside's own 70.2% is footnoted "in Poolside's agent harness."
 - **Recovery/loops:** clean on held-out work; 0/12 injection-in-tool-output executed (resisted). **Long regime: thinking-OFF completed a 30/30-turn persistent-failure agent loop cleanly; thinking-ON hung at turn 11/30 and never returned (~91 min). Keep thinking OFF for any long-running agent, doubly so given Poolside now ships thinking on-by-default with the token cap dropped (see changelog).**
 
@@ -205,7 +205,29 @@ What each dataset actually supports, kept separate:
 
 **What this means for the guide overall:** the persona lever is now three-stack confirmed and is the one to rely on. The task-shape claim is weakened. And the "near-zero thinking in a real pipeline" line in §5d needs the length caveat above attached to it.
 
-## 5f. The failure that silently kills agent loops (@Defilan)
+## 5f. Fourth stack: thinking is regime-dependent, not simply bad (@apollo-mg)
+
+[@apollo-mg](https://github.com/apollo-mg) measured thinking on **HumanEval+, 492 samples**, and got the opposite sign to our behavioral battery. Both results are real; they are measuring different regimes, and this is the most important nuance in the guide.
+
+| Regime | Winner | Evidence |
+|---|---|---|
+| **Single-turn code generation** with a verifiable answer | **thinking ON** | 90.85% vs 88.21% (+2.64 pts), and flaky problems **halve** (11 vs 24). Thinking stabilizes output, it does not only add accuracy |
+| **Held-out behavioral / integrity** axes | **thinking OFF** | 94.2% vs 91.3%; ON fabricates bugs in clean code, over-refuses an authorized pentest, and capitulates to a planted false premise (8-9 down to 1-3) |
+| **Long agentic loops** | **thinking OFF** | OFF completed 30/30, ON hung at 11/30; and `false` eliminates truncated-think entirely (0/15 cap-hits vs 2/15) |
+
+**So the cost of our own top recommendation is now quantified, and we should own it:** turning thinking off costs about **2.6 points and doubles flakiness** on single-turn codegen. That is a real price this guide was silently asking readers to pay. Use the regime table, not a blanket "off."
+
+**Two methodology findings from that run that apply to anyone benchmarking this model:**
+- **Treat cap-hits as failures, not truncations.** p95 output is 10,152 tokens and median 1,945, so **~12k is a better ceiling than 16k**. Crucially, raising the budget does **not** convert cap-hitters into passes: compression-ratio analysis showed most are **degeneration loops**, which more budget cannot save. Counting them as "needs more tokens" is what manufactures false "the model cannot do this" conclusions. **This indicts our own harness**, whose arms retried to 16k and 64k ceilings on exactly that assumption (§2 confound).
+- **A loop-detection stopping rule is the highest-value harness change available.** 11 of 492 samples produced no extractable answer, and that termination failure accounts for ~87% of their gap to a larger model. Compression ratio cleanly separates degeneration loops from coherent long reasoning, so a harness that aborts on a loop signature and retries with thinking off should recover most of them. That transfers to any model with this failure mode.
+
+**⚠️ Do not carry the ~12k ceiling straight into an agent loop** (raised by [@Blackwellboy](https://github.com/Blackwellboy) on this PR). The 12k figure comes from the p95 of **single-turn** outputs, and the two recommendations interact badly: a *tighter* cap makes a turn more likely to run out of budget **inside** the reasoning block, which is exactly the §5f wire failure. In a benchmark that just costs you one scored sample. In an agent loop it returns an assistant message with reasoning but no content and no `tool_calls`, the server rejects it, and **every subsequent turn fails identically**, silently, with no diff to show for it. So split the advice by regime:
+- **Single-turn / benchmark:** ~12k is right, and score cap-hits as failures.
+- **Agent loop:** keep more headroom, and separately make the loop survive a capped turn (drop the malformed message or substitute a placeholder) rather than relying on the cap never being hit. Turning thinking off with the kwarg removes the failure at the source: 0/15 cap-hits versus 2/15.
+
+**Scope, stated as they stated it:** one benchmark, single-turn codegen only, two sampling points (t0.7 thinking-on, t0.6 thinking-off), no temperature or top_p/top_k/min_p sweep, and `-fa on/off` untested on Laguna with server flags unrecorded. Nobody should publish "optimal sampling for Laguna" from this, including us. The long-agentic regime where the contested behavior lives is untouched by their run.
+
+## 5g. The failure that silently kills agent loops (@Defilan)
 
 Worth its own section because it is silent, fatal, deterministic, and reads as model incapability rather than a wire-format bug.
 
@@ -221,7 +243,7 @@ llama.cpp rejects that with `400: Assistant message must contain either 'content
 - **`enable_thinking: false` also eliminates it at the source:** no think block means no truncated think means no poisoned conversation (0/15 cap-hits vs 1/15 and 2/15).
 - **Relevant to our own numbers:** our CAPPED arm pinned 4,096 with no retry, precisely the regime that produces truncated thinks, and this is a candidate mechanism for our thinking-ON long-agentic hang at turn 11.
 
-## 5g. Serving config measured on gfx1151 / Strix Halo (@Defilan)
+## 5h. Serving config measured on gfx1151 / Strix Halo (@Defilan)
 
 ```
 --jinja --no-mmap --flash-attn on --spec-type none