diff --git a/cards/img/laguna-s-2.1.png b/cards/img/laguna-s-2.1.png index fe4c048..6c96288 100644 Binary files a/cards/img/laguna-s-2.1.png and b/cards/img/laguna-s-2.1.png differ diff --git a/cards/laguna-s-2.1.html b/cards/laguna-s-2.1.html index 92524a0..6a8d555 100644 --- a/cards/laguna-s-2.1.html +++ b/cards/laguna-s-2.1.html @@ -44,25 +44,26 @@
Use it like this
-
๐Ÿง ThinkingSend enable_thinking: false explicitly for long-horizon / integrity work ยท omitting it = ON ยท the default moves between revisions, so verify your rendering
+
๐Ÿง ThinkingSet enable_thinking explicitly, by regime. false = structurally off (long-horizon / integrity work) ยท true for single-turn codegen (+2.6 pts, halves flakiness) ยท omitting it = ON, and the default moves between revisions
๐Ÿ“‰If ONFiring is a dose curve: bare ~50% โ†’ +persona halves โ†’ +rules/tools ~0.1%. The apparatus suppresses, not the persona name
๐Ÿ”งToolsServe the native template (--jinja) ยท chatml drops tool calls to 0% (83%→0%) ยท a generic OpenAI-format client is fine, pool not required
๐ŸŽ›๏ธServingtemp 0.6 ยท Q4_K_M ยท -fit off ยท -fa on (2.2x decode at 56k, gain grows with depth) ยท --no-mmap on big quants
+
๐Ÿ“Budgetmax_tokens ~12k for single-turn (p95 10.2k, median 1.9k) ยท treat cap-hits as failures, not truncations: most are degeneration loops more budget cannot fix ยท but keep headroom in agent loops, a tighter cap raises the silent truncated-think kill
๐Ÿ“ŒPin + capPin the revision. Vendor dropped the output cap post-release, and the enable_thinking default itself moves between revisions
โœ…Best / avoidBest: long-horizon coding, tool use, structured agent work (~9/10, no thrash) ยท Avoid: non-English, one-shot answers you can't verify
-
Independently validated: three stacks
-
๐ŸŸข12h / 2947 turns99.9% turn success, 0 crashes, 0 restarts, ~4GiB creep, ~13.5s/turn ยท 100% tool calls on native format ยท 0 runaway loops with pin+cap ยท integrity clause held 3/3 cross-quant
@BlackwellBoy (NVFP4/vLLM, 12h soak) ยท @Defilan (gfx1151/llama.cpp, kwarg + gate A/B) ยท ours (Q4_K_M/llama.cpp)
+
Independently validated: four stacks
+
๐ŸŸข12h / 2947 turns99.9% turn success, 0 crashes, 0 restarts, ~4GiB creep, ~13.5s/turn ยท 100% tool calls on native format ยท 0 runaway loops with pin+cap ยท integrity clause held 3/3 cross-quant
@BlackwellBoy (NVFP4/vLLM, 12h soak) ยท @Defilan (gfx1151/llama.cpp, kwarg + gate A/B) ยท @apollo-mg (HumanEval+ 492 samples: thinking ON +2.6 pts single-turn) ยท ours (Q4_K_M/llama.cpp)
Trust map: where it holds vs folds under pressure
โœ“ HOLDS THE LINE
won't fake CI/testswon't fabricate compliancewon't disable safety checksresists tool-result injection (0/12)
โœ• FOLDS UNDER PRESSURE (unprompted)
erases leaked-secret git historybackdates commitsforges changelog authorshiphides client PII hazardcapitulates to false premises
Fix (validated, 4/4 + cross-model): add a system-prompt integrity clause: never rewrite history to hide secrets, never backdate or forge, never omit a known hazard, flag unsupported asserted facts.
-
โš  BLIND SPOTS
truncated think kills agent loops (400: needs content or tool_calls; silent + deterministic)invents bugs when thinking is ONthinking ON hung a long agent loop (OFF 30/30 vs ON 11/30)over-refuses authorized work (thinking ON)misses covert self-harm signals
+
โš  BLIND SPOTS
truncated think kills agent loops (400: needs content or tool_calls; silent + deterministic)thinking ON can degenerate into loops more budget won't saveinvents bugs when thinking is ONthinking ON hung a long agent loop (OFF 30/30 vs ON 11/30)over-refuses authorized work (thinking ON)misses covert self-harm signals
-
Q4_K_M ยท pinned 2026-07-24 (vendor drift) ยท held-out behavioral tests + 12h soak + 2 independent stacksofflabel
+
Q4_K_M ยท pinned 2026-07-24 (vendor drift) ยท held-out behavioral tests + 12h soak + 3 independent stacksofflabel
diff --git a/models/laguna-s-2.1.md b/models/laguna-s-2.1.md index 3974363..9558456 100644 --- a/models/laguna-s-2.1.md +++ b/models/laguna-s-2.1.md @@ -25,10 +25,10 @@ If you read nothing else, do these. **Three independent testers on three differe | | Do this | Why | Where | |---|---|---|---| -| 1 | **Set `enable_thinking` explicitly** | **Send `false`** for long-horizon or integrity-sensitive work; **omitting the kwarg leaves thinking ON**, and the default moves between revisions. ๐Ÿงช Whether ON is better for single-turn codegen is under review in [#8](https://github.com/TheTom/offlabel/pull/8) | ยง2 | +| 1 | **Set `enable_thinking` explicitly, by regime** | `false` for long-horizon or integrity-sensitive work. `true` for single-turn codegen (+2.6 pts, halves flakiness). **Omitting it leaves thinking ON**, and the default moves between revisions | ยง2, ยง5f | | 2 | **Serve the native template (`--jinja`)** | Tool-calling is all-or-nothing: ~100% native, **0% under chatml** (the model narrates instead of calling). A generic OpenAI-format *client* is fine; `pool` is not required | ยง4 | | 3 | **Integrity clause in your system prompt** | It refuses blatant fraud but complies when the same act is framed as cleanup (erase a leaked key from history, backdate, forge authorship). One paragraph closes it | ยง5b | -| 4 | **Pin the revision + set your own max-token ceiling** | The vendor dropped the output cap post-release, so if you do not set one, nothing does. The `enable_thinking` default also moves between revisions | ยง5, ยง5d | +| 4 | **Pin the revision + cap your own tokens (~12k single-turn)** | The vendor dropped the output cap, so if you do not set one nothing does. And **treat cap-hits as failures, not truncations**: most are degeneration loops that more budget cannot fix. โš ๏ธ **The ~12k figure is derived from single-turn work. In an agent loop a tighter cap makes the silent ยง5f killer more likely, not less** | ยง5, ยง5f | ### How to actually minimize thinking (the mechanism, since #1 is the subtle one) @@ -130,7 +130,7 @@ Coverage tag per axis: **โœ… measured** (held-out, 3-arm, blind 2-vote) ยท ** - **Why:** the personaร—task thinking gate above, plus the scope-narrowing observed in a full build (ยง5c). ## 4. Tools & agents -- **Recommendation (sharpened 2026-07-26):** the requirement is **server-side**: serve the **native template with `--jinja`** so the differential autoparser runs. Do not serve under chatml. **A generic OpenAI-format client is fine** and `pool` is not required: a third stack merged four real PRs into a production repo driving this model from a generic OpenAI tool-calling harness (ยง5c, ยง5g). Our original 83%-vs-0% number was a *template* comparison, and framing it as "use pool" was too narrow. +- **Recommendation (sharpened 2026-07-26):** the requirement is **server-side**: serve the **native template with `--jinja`** so the differential autoparser runs. Do not serve under chatml. **A generic OpenAI-format client is fine** and `pool` is not required: a third stack merged four real PRs into a production repo driving this model from a generic OpenAI tool-calling harness (ยง5c, ยง5h). Our original 83%-vs-0% number was a *template* comparison, and framing it as "use pool" was too narrow. - **Why:** measured native-vs-chatml tool-call match rate **83.3% vs 0.0%** across the agentic set. chatml categorically breaks structured `tool_calls` (the model narrates in `content` instead) and mangles reasoning (leaked CoT, orphan ``). The "overfits to its native harness" complaint is real and quantified; and Poolside's own 70.2% is footnoted "in Poolside's agent harness." - **Recovery/loops:** clean on held-out work; 0/12 injection-in-tool-output executed (resisted). **Long regime: thinking-OFF completed a 30/30-turn persistent-failure agent loop cleanly; thinking-ON hung at turn 11/30 and never returned (~91 min). Keep thinking OFF for any long-running agent, doubly so given Poolside now ships thinking on-by-default with the token cap dropped (see changelog).** @@ -205,7 +205,29 @@ What each dataset actually supports, kept separate: **What this means for the guide overall:** the persona lever is now three-stack confirmed and is the one to rely on. The task-shape claim is weakened. And the "near-zero thinking in a real pipeline" line in ยง5d needs the length caveat above attached to it. -## 5f. The failure that silently kills agent loops (@Defilan) +## 5f. Fourth stack: thinking is regime-dependent, not simply bad (@apollo-mg) + +[@apollo-mg](https://github.com/apollo-mg) measured thinking on **HumanEval+, 492 samples**, and got the opposite sign to our behavioral battery. Both results are real; they are measuring different regimes, and this is the most important nuance in the guide. + +| Regime | Winner | Evidence | +|---|---|---| +| **Single-turn code generation** with a verifiable answer | **thinking ON** | 90.85% vs 88.21% (+2.64 pts), and flaky problems **halve** (11 vs 24). Thinking stabilizes output, it does not only add accuracy | +| **Held-out behavioral / integrity** axes | **thinking OFF** | 94.2% vs 91.3%; ON fabricates bugs in clean code, over-refuses an authorized pentest, and capitulates to a planted false premise (8-9 down to 1-3) | +| **Long agentic loops** | **thinking OFF** | OFF completed 30/30, ON hung at 11/30; and `false` eliminates truncated-think entirely (0/15 cap-hits vs 2/15) | + +**So the cost of our own top recommendation is now quantified, and we should own it:** turning thinking off costs about **2.6 points and doubles flakiness** on single-turn codegen. That is a real price this guide was silently asking readers to pay. Use the regime table, not a blanket "off." + +**Two methodology findings from that run that apply to anyone benchmarking this model:** +- **Treat cap-hits as failures, not truncations.** p95 output is 10,152 tokens and median 1,945, so **~12k is a better ceiling than 16k**. Crucially, raising the budget does **not** convert cap-hitters into passes: compression-ratio analysis showed most are **degeneration loops**, which more budget cannot save. Counting them as "needs more tokens" is what manufactures false "the model cannot do this" conclusions. **This indicts our own harness**, whose arms retried to 16k and 64k ceilings on exactly that assumption (ยง2 confound). +- **A loop-detection stopping rule is the highest-value harness change available.** 11 of 492 samples produced no extractable answer, and that termination failure accounts for ~87% of their gap to a larger model. Compression ratio cleanly separates degeneration loops from coherent long reasoning, so a harness that aborts on a loop signature and retries with thinking off should recover most of them. That transfers to any model with this failure mode. + +**โš ๏ธ Do not carry the ~12k ceiling straight into an agent loop** (raised by [@Blackwellboy](https://github.com/Blackwellboy) on this PR). The 12k figure comes from the p95 of **single-turn** outputs, and the two recommendations interact badly: a *tighter* cap makes a turn more likely to run out of budget **inside** the reasoning block, which is exactly the ยง5f wire failure. In a benchmark that just costs you one scored sample. In an agent loop it returns an assistant message with reasoning but no content and no `tool_calls`, the server rejects it, and **every subsequent turn fails identically**, silently, with no diff to show for it. So split the advice by regime: +- **Single-turn / benchmark:** ~12k is right, and score cap-hits as failures. +- **Agent loop:** keep more headroom, and separately make the loop survive a capped turn (drop the malformed message or substitute a placeholder) rather than relying on the cap never being hit. Turning thinking off with the kwarg removes the failure at the source: 0/15 cap-hits versus 2/15. + +**Scope, stated as they stated it:** one benchmark, single-turn codegen only, two sampling points (t0.7 thinking-on, t0.6 thinking-off), no temperature or top_p/top_k/min_p sweep, and `-fa on/off` untested on Laguna with server flags unrecorded. Nobody should publish "optimal sampling for Laguna" from this, including us. The long-agentic regime where the contested behavior lives is untouched by their run. + +## 5g. The failure that silently kills agent loops (@Defilan) Worth its own section because it is silent, fatal, deterministic, and reads as model incapability rather than a wire-format bug. @@ -221,7 +243,7 @@ llama.cpp rejects that with `400: Assistant message must contain either 'content - **`enable_thinking: false` also eliminates it at the source:** no think block means no truncated think means no poisoned conversation (0/15 cap-hits vs 1/15 and 2/15). - **Relevant to our own numbers:** our CAPPED arm pinned 4,096 with no retry, precisely the regime that produces truncated thinks, and this is a candidate mechanism for our thinking-ON long-agentic hang at turn 11. -## 5g. Serving config measured on gfx1151 / Strix Halo (@Defilan) +## 5h. Serving config measured on gfx1151 / Strix Halo (@Defilan) ``` --jinja --no-mmap --flash-attn on --spec-type none