Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions models/laguna-s-2.1.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ If you read nothing else, do these. **Three independent testers on three differe

| | Do this | Why | Where |
|---|---|---|---|
| 1 | **Minimize thinking** (see the how, below) | It is net-negative on held-out work. **Do not just send `enable_thinking: false`, that is a no-op** | §2 |
| 1 | **Minimize thinking** (see the how, below) | It is net-negative on held-out work. **Send `enable_thinking: false` explicitly; omitting the kwarg leaves thinking available** | §2 |
| 2 | **Native `poolside_v1` tool format** | Tool-calling is all-or-nothing: ~100% native, 0% under a generic/chatml format (the model narrates instead of calling) | §4 |
| 3 | **Integrity clause in your system prompt** | It refuses blatant fraud but complies when the same act is framed as cleanup (erase a leaked key from history, backdate, forge authorship). One paragraph closes it | §5b |
| 4 | **Pin the revision + set your own max-token ceiling** | Post-release config drift turned thinking on by default and dropped the output cap, which is exactly the "it never stops" recipe. If you do not set a ceiling, nothing does | §5, §5d |
Expand All @@ -34,7 +34,7 @@ If you read nothing else, do these. **Three independent testers on three differe

There is no single clean off-switch. Ranked by how well each is evidenced:

1. **Never send `enable_thinking: true`.** That is the one setting that demonstrably flips the template from a pre-closed `</think>` to an open `<think>`. Sending `false` changes nothing (it is already the default), so if that was your whole "thinking is off" story, it was a no-op.
1. **Send `enable_thinking: false` explicitly.** This is the one structural off-switch: `false` renders a pre-closed `</think>` and thinking becomes impossible (0/15 reasoned across three prompt shapes at a fixed budget, [#5](https://github.com/TheTom/offlabel/issues/5)). **Omitting the kwarg does not give you the `false` path**: on the tested llama.cpp stack the absent arm renders byte-identical to `true` (the server supplies the value, so the template's `default(false)` never runs), and the model then reasons roughly half the time. The previous version of this advice said sending `false` changes nothing; that was wrong, see §2.
2. **Put a named professional identity in your system prompt** ("You are a senior staff engineer."). This is the lever with the most evidence behind it: three stacks, and in single-turn probes it drove visible reasoning to a clean zero (§5e: 6/6 fired without one, 0/5 with one).
3. **Expect attenuation, not elimination, in long agentic loops.** The same persona that zeroed single-turn thinking only *shortened* it across a 20+ turn tool-using loop (§5e). Budget for some reasoning in long runs regardless.
4. **Set your own max-output-token ceiling** as the backstop, since the vendor dropped theirs (§5).
Expand Down Expand Up @@ -90,11 +90,11 @@ Coverage tag per axis: **✅ measured** (held-out, 3-arm, blind 2-vote) · **

## 2. Thinking / reasoning
- **Recommendation:** default **OFF**. Enable ON only for an isolated hard-reasoning turn, and not on integrity-sensitive work.
- **Control, precisely (corrected 2026-07-25, thanks [@Defilan](https://github.com/Defilan), see §5e):** the `enable_thinking` chat-template kwarg **defaults to `false`**, and in that branch the template hands the model a **pre-closed `</think>`**; with `true` it hands an **open `<think>`**:
- **Control, precisely (corrected again 2026-07-26 by [@Defilan](https://github.com/Defilan)'s fixed-budget A/B in [#5](https://github.com/TheTom/offlabel/issues/5), superseding the 07-25 correction):** `enable_thinking: false` is a **real, structural off-switch**. It renders a pre-closed `</think>` and the model cannot reason past it: 0/15 reasoned across three prompt shapes at a fixed `max_tokens`. **Omitting the kwarg is not the `false` path**: on the tested llama.cpp stack the absent arm renders **byte-identical to `true`** (open `<think>`), because the server supplies the value itself and the template's `default(false)` branch never runs. In that arm the model reasons roughly half the time (8/15 absent, 9/15 explicit `true`). Verified prompt-level via `/apply-template`, deterministically and with no sampling:
```jinja
{%- if enable_thinking -%}{{- '<think>' -}}{%- else -%}{{- '</think>' -}}{%- endif -%}
```
**So passing `enable_thinking: false` explicitly does nothing** (it is already the default). If you thought you disabled thinking by sending `false`, you changed no bytes. Passing `true` *is* a real change. The lever that actually moves behavior on a default config is the **persona** (below), not the kwarg.
**On top of that, which arm "absent" lands in is revision-dependent:** [@BlackwellBoy](https://github.com/Blackwellboy)'s NVFP4 rev `0761412` checkpoint ships an on-disk template that defaults `enable_thinking` to **`true`** outright, as does Poolside's current HF upload (see the config-drift changelog, §10; details in [his lab's README](https://github.com/Blackwellboy/laguna-s21-lab)). So do not reason about thinking from any template's default, on any stack: **send the kwarg explicitly, and verify your own rendering** (the `render_check.py` proposed in [#6](https://github.com/TheTom/offlabel/pull/6) does this in one command). The persona (below) remains a real second lever, and the only one available when your serving path ignores the kwarg.
- **Firing gate:** a **persona×task conjunction**: a named professional identity ("senior staff engineer") suppresses reasoning to zero. Reproduced independently on a third stack (§5e: 6/6 fired with no persona, 0/5 with one). The *task* half is less settled: we saw coding-shaped tasks suppress regardless of persona; @Defilan's coding probe fired 6/6 without a persona, so the boundary of "coding-shaped" is not pinned down. Treat **persona as the reliable lever** and task-shape as unresolved.
- **⚠️ Open question on what "off" actually means.** Our firing numbers were measured by reading the `reasoning_content` field. Because our OFF arm sent `enable_thinking: false` (= the default, pre-closed `</think>`), **"0% fired" cannot distinguish "the model did not reason" from "it reasoned but nothing populated `reasoning_content`."** On @Defilan's stack, where `reasoning_content` is reliably populated, the model **produces reasoning anyway** under that same default. So the mechanism label may be wrong even though the behavior difference is real: the OFF and ON arms differed materially on outcomes (pass rate, failure modes, and 30/30 vs hung-at-11/30), so *something* changes. Pinning down what needs a same-stack A/B that reads `reasoning_content` and raw content separately. **Not yet run; treat the firing-rate row below as a parsed-reasoning rate, not a proven model-choice rate.**
- **Dose-response** (3 arms, same persona, held-out battery):
Expand Down Expand Up @@ -221,6 +221,7 @@ What each dataset actually supports, kept separate:
- **Scope caveats:** single test window, one quant, config is a moving target (version-pinned). Multilingual/crisis probes small-n. Mitigation validated thinking-OFF; thinking-ON re-run and long-regime non-termination pending. Benchmarks not reproduced (behavior-vs-behavior, not a leaderboard).

## Changelog
- `2026-07-26`: §2 control corrected AGAIN, per [@Defilan](https://github.com/Defilan)'s fixed-budget A/B in [#5](https://github.com/TheTom/offlabel/issues/5), which supersedes the 07-25 correction. `false` is a real structural off-switch (pre-closed `</think>`, 0/15 reasoned); the **absent** arm renders byte-identical to `true` on the tested llama.cpp stack (the server supplies the kwarg, so the template's `default(false)` never runs) and reasons about half the time. The 07-25 line "passing `false` explicitly does nothing" was wrong. Quickstart #1 and the minimize-thinking ranking now say: send `false` explicitly. Also added: the default itself is revision-dependent, [@BlackwellBoy](https://github.com/Blackwellboy)'s NVFP4 rev `0761412` checkpoint and Poolside's current HF upload both default `enable_thinking` to `true`, consistent with the §10 config-drift note. Not yet re-folded here: #5's re-measured persona-gate magnitude (10/18 vs 1/18, artifact-corrected baseline) and the restored task-shape signal; those await the maintainer's read.
- `2026-07-24`: initial behavioral assessment + serving findings on Q4_K_M; §5b integrity clause validated (4/4 caves closed, thinking-OFF).
- `2026-07-25`: added §5e third-stack field notes from @Defilan (gfx1151/llama.cpp/generic harness, `reasoning_content` reliably populated) and corrected §2 as a result. The `enable_thinking` kwarg **defaults to false**, so passing `false` is a no-op against the template default; verified in our own fork's template. Persona gate reproduced (6/6 fired with no persona, 0/5 with one); the **task-shape** half did NOT reproduce and is now marked unresolved. NEW: the gate **attenuates with conversation length** (single-turn persona drove thinking to zero; a 20+ turn agentic loop only shortened it, 35,913 to 20,573 chars, block count 10 to 9), which a scenario battery cannot surface. Flagged an open limitation of our own firing numbers: they were read from `reasoning_content`, so "0% fired" cannot distinguish not-reasoning from not-parsed. §5b gains his independent rule-by-rule ablation on freshly-worded scenarios (clause reproduces 7/7 vs 3/7 unprompted; rule (1) load-bearing; floor of (1)(2)(4)) and his walk-back of the rule-(3) cost magnitude.
- `2026-07-25`: applied @BlackwellBoy's two precision corrections to §5d after review. (a) His ~0.1% thinking rate is an OBSERVED routing rate on that rev/serving path, not proof the model declined to think: empty `reasoning` fields came back even when thinking was requested, so part of it may be template/parser-level. I had written that his number confirmed our persona-gate mechanism; that was overreach and is now corrected. (b) "Zero loops" is scoped to zero UNBOUNDED-GENERATION loops while the gate barely opened; his 9 logged incidents were his own driver's session cap firing by design, so it is not evidence about thinking-heavy workloads. Also recorded his stack precisely (NVFP4 on vLLM, vs our Q4_K_M on llama.cpp) and linked his published raw logs. Separately added the flash-attention serving finding (`-fa on`, up to 2.2x decode at 56K depth) from an independent Strix Halo test.
Expand Down