A/B results, a correction to our own §2 claim, and four things from our stack

[Defilan]

Thanks for merging #3, and for reading it rather than rubber-stamping it. Also
for disclosing the token-budget difference across your three arms. That was not
something I would have found, and it is the kind of thing most people quietly
fix rather than write down.

Ran the A/B. Your spec was hold max_tokens fixed across arms and capture
reasoning_content and raw content separately, and that is what these runs do:
a single max_tokens=1400 for every arm and every cell, no retry, no per-arm
ceiling. A truncated response is recorded as truncated rather than re-requested.

One thing first, because it changes what §2 should say.

Our own §2 claim was wrong

The enable_thinking finding I sent in #2 does not hold. It is the one you
corrected §2 for. The correct version is close to the opposite: false is the
working lever, and omitting the kwarg is what gives you thinking.

What our first measurement got wrong

Worth passing on, because you run the same kind of experiment and the failure is
invisible from the outside.

Our first pass ran arms in blocks, all of A then all of B, on one repeated
prompt, with the prompt cache on. That combination produces results that
look clean and are not:

• Blocked arms mean anything drifting over the run (thermal, cache state, slot
  reuse) lands entirely on whichever arm was running then. Arm and time are not
  separable.
• One repeated prompt with the cache on collapses the very variance you are
  measuring. Five identical requests approximate one request sampled five times.

It produced 5/5 in one arm and 0/5 in another. That reads as a strong effect. It
was mostly an artifact, and staring at the numbers would never have caught it,
because manufactured consistency looks exactly like a real result.

What we changed:

	first pass	now
arm order	blocked	interleaved round robin
prompts	1	3 shapes
prompt cache	on	cache_prompt: false plus a unique nonce per request
token budget	varied	fixed at 1400 for every arm, no retry
control cell	none	an arm where the effect is structurally impossible
prompt-level check	none	/apply-template, deterministic, no sampling

The control cell is the one I would push hardest: an arm where the thing you
are measuring cannot physically happen, so if it shows up anyway the instrument
is broken rather than the finding being strong. The prompt-level check turns
out to settle this entire question on its own, with no statistics at all.

The underlying habit, which is the real lesson: we reasoned from template source
instead of testing rendered output. Reading is not measuring.

Data 1: the rendered prompt

/apply-template returns the formatted prompt without generating, so this is
deterministic:

request	rendered tail
no chat_template_kwargs	<assistant><think>
chat_template_kwargs: {}	<assistant><think>
{"enable_thinking": true}	<assistant><think>
{"enable_thinking": false}	<assistant></think>

Byte-comparing the full strings: absent == true is True, absent == false is
False. Omitting the kwarg is not the false path, it is the true path. An empty
kwargs object behaves like omitting it, so it is the enable_thinking key
specifically, not the presence of the field.

Data 2: why, template versus server

Your template in stock jinja2, against the same inputs through the server:

	undefined / absent	explicit false	explicit true
pure jinja2	</think>	</think>	<think>
llama.cpp	<think>	</think>	<think>

Your | default(false) is correct and behaves exactly as written. The server
overrides it, so that branch is unreachable in practice. This was never a
template bug or a misreading of one. It was an assumption that the template gets
the last word.

Flagged as inference, since I have not read the llama.cpp source: that the
server supplies the value itself is the explanation. What is proven is that
the two renderings differ.

This transfers to you: we serve your exact template, sha1 8b0c10f2febc,
3788 bytes, byte-identical to poolside-Laguna-S-2.1.jinja at 3e7bbe1.
--jinja, no --reasoning-budget set.

Data 3: behaviour

45 requests. Three prompt shapes, interleaved, cache_prompt: false, unique
nonce, temp 0.6, max_tokens=1400 for every arm.

arm	reasoned	mean reasoning chars	hit token cap
absent	8/15	1316	1/15
false	0/15	0	0/15
true	9/15	1863	2/15

false against absent: one-sided p = 0.001. absent against true: p = 0.5,
which is what byte-identical prompts should give.

false makes thinking structurally impossible, since the tag closes before
generation starts. absent and true make it available, and the model then
chooses per request. "Absent" is not a neutral default, it is thinking-on.

What this settles and what it does not

Taking your separation, which is the right one:

• Settled, and it generalizes: a pre-closed </think> does stop the model
  reasoning. 0/15 with zero characters, across three prompt shapes, at a fixed
  budget. That is a model-behavior claim, not a serving-path artifact.
• Not settled: whether your specific 0% was a parsing artifact. Only your
  stack answers that, and this cannot substitute for it.

One extra data point for the parsing question, though it does not replace your
re-run: zero of 45 responses had reasoning in content but nothing in
reasoning_content. On this stack the field is faithful, so did-not-reason and
was-not-parsed do not diverge here.

Data 4: the persona gate, re-measured

The same blocked-and-cached flaw inflated our 6/6 versus 0/5. Re-ran crossed
against the kwarg, 72 requests, same handling, same fixed budget:

system prompt	enable_thinking	reasoned	mean reasoning chars
none	absent	10/18	1471
You are a senior staff software engineer.	absent	1/18	72
none	false	0/18	0
persona	false	0/18	0

The gate is real: 10/18 against 1/18, one-sided p = 0.0014. The baseline was
wrong, not the conclusion. Roughly 56 percent to 6 percent rather than 100 to
zero, so §5e is right that a gate exists and overstates how sharp it is.

The bottom two rows are the control and the reason I believe the top two: with
thinking structurally unavailable, the persona changes nothing, 0/18 either way.

It also matches your attenuation point. A persona takes thinking to 1/18, not
0/18. It shortens rather than stops, in single turns as well as long loops.

Data 5: task shape, which you had right

We talked you out of this one and the better data points back at you:

prompt shape	no persona	with persona
short-factual	6/6	0/6
code-refactor	2/6	1/6
design-question	2/6	0/6

Coding-shaped reasoned least, 2/6 with no persona at all, while a
one-sentence factual question reasoned 6/6. That is what you originally
described before §2 was weakened on our evidence.

It explains our counterexample directly: our Go refactor fired 6/6 then, and the
same shape gives 2/6 under a clean run. Same artifact. Treat our 6/6 as noise,
not as evidence against you.

Six per cell, so not proof, and I am not asking you to flip it back on my
say-so. But the evidence that weakened it was worse than what you already had.

What we did not re-measure

• The agentic attenuation numbers (35,913 to 20,573 chars, 10 blocks to 9)
  came from one pair of real runs, not the probe harness. Not artifact-driven
  the same way, but n=1 pair. I would not lean on the 43 percent.
• The §5b rule ablation is unaffected: different axis, seven distinct
  prompts rather than one repeated, cross-checked by an independent judge at 93
  percent agreement plus the Qwen replication. Not re-run, no reason to doubt it.

---

Four things from our stack that worked

Separate from the correction. All measured here, all things your readers will
plausibly hit.

1. A reasoning-only turn that gets truncated will kill an agent loop

This is the one I would most want in the guide, because it is silent and fatal
and it is not obvious.

If a thinking model hits a per-turn token cap while still inside <think>,
it returns a message with reasoning_content populated and no content and no
tool_calls. A harness that preserves that turn so the model can resume then
sends it back. Both fields are omitempty in the OpenAI schema, so it
serializes as:

{"role":"assistant","reasoning_content":"...partial think..."}

llama.cpp rejects that with 400: Assistant message must contain either 'content' or 'tool_calls'!. Every subsequent turn fails the same way. The run
dies with no diff, and it presents as model incapability rather than a wire
format bug.

It killed 5 of 14 of our coder runs before we found it, and it reproduced at
the same turn number across two independent runs, which is what finally gave
it away: it is deterministic, not flaky.

The fix is one line of principle: never let an assistant message reach the wire
with neither content nor tool_calls. Substitute a placeholder for the empty
content and keep the reasoning.

Why it is relevant to you specifically: your CAPPED arm pinned 4,096 with no
retry, which is exactly the regime that produces truncated thinks. Any harness
that preserves a truncated reasoning turn will hit this, not just ours.

And it is the practical reason the false column above matters:
enable_thinking: false eliminated truncation entirely, 0/15 against 1/15 and
2/15. No think, no truncated think, no poisoned conversation.

2. DFlash is real but workload-sensitive to an extreme

poolside ships a drafter in the same GGUF repo,
laguna-s-2.1-DFlash-BF16.gguf, 1.11B params, about 2.1GB. No conversion
needed. It requires flash attention on.

-md /path/to/laguna-s-2.1-DFlash-BF16.gguf --spec-type draft-dflash --spec-draft-n-max 15

workload	draft acceptance	decode
real code editing	53.8%	1.15x (27.53 to 31.63 tok/s)
synthetic random text	0 to 8%	2.1x slower

The trap: benchmark it on synthetic filler and you will conclude it is broken,
because every rejected draft is wasted compute. Speculation predicts the model's
own output, so it only pays when that output is predictable, which for a coding
model means editing code already in context. It also costs 2.1GB of unified
memory the KV cache would otherwise use. Depth behaviour on realistic content is
unmeasured on our side.

3. Generic harness update to §5c: four merged, one caught by review

We originally gave you n=2. Current record with a generic OpenAI tool-calling
harness, not pool, on real backlog issues in a production repo:

• Four merged: shared phase constants across two packages, a prompt-depth
  benchmark sweep, a prompt-cache fix, and a status-condition fix. All passed a
  full gate (fmt, vet, lint, package tests including a 249s envtest suite,
  codegen drift, plus a bite check that reverts the production change and
  re-runs the new tests to confirm they fail).
• One gate-passed and I rejected it on review. The change gated its work on
  git diff HEAD~1 HEAD inside a --depth 1 clone, where HEAD~1 does not
  exist. It failed silently, so the feature was a no-op that printed a header
  and passed. Its tests asserted the rendered template contained the string
  git diff --name-only HEAD~1 HEAD, which pins the bug rather than testing
  behaviour.

I am including the failure deliberately. Four for five is a fine rate, but the
useful part is the shape of the miss: the gate could not catch it because the
tests were strings.Contains assertions on rendered text rather than exercises
of the logic. That is a harness lesson rather than a model one, and it is the
same lesson as our classifier: a check that cannot fail proves nothing.

4. Serving config that is actually stable on gfx1151

For anyone landing on Strix Halo, measured rather than assumed:

--jinja --no-mmap --flash-attn on --spec-type none
--cache-type-k f16 --cache-type-v f16 --ctx-size 131072

• --jinja is load-bearing: without it the differential autoparser never runs
  and tool calls stop parsing.
• --no-mmap because the Q4_K_M weights exceed the mmap-friendly window.
• Flash attention on is worth 2.27x prefill and 2.22x decode at 56k, and
  decode retention at depth goes from 40.0% to 75.8% of the shallow baseline.
  The gain grows with depth, which is the opposite of the usual expectation.
• f16 KV plus amdgpu.lockup_timeout=20000 on the host is what made long
  sessions stable for us.

Scripts and raw data

#3 is merged, so I will open a second small PR with both scripts: the
deterministic /apply-template comparison and the interleaved behavioural run
with the control cell. Raw JSON for all 117 requests, with full response bodies
and reasoning traces, is yours either in that PR or attached here, whichever you
prefer.

[TheTom]

Closing: everything in here landed.

Your §2 correction (false is the working lever, omitting the kwarg gives you thinking) is merged as the primary §2 statement in PR #7, along with the revision-dependent-default caveat. The /apply-template render check is what settled it, and the guide now tells readers to run that check rather than read the template, with your "reading is not measuring" framing kept intact.

Your harness is merged as scripts/thinking-probes/ in PR #6, render_check.py included.

The methodology corrections (interleaved arms not blocked, three prompt shapes, cache_prompt: false plus per-request nonce, fixed token budget, a control cell where the effect is structurally impossible) are now written into the guide's verify-your-setup block and the probe READMEs. The control-cell point in particular generalized past Laguna and is in patterns.md.

The token-budget confound in our own three arms stays disclosed in §2 and is tracked as an open re-run in ROADMAP.md §5, unresolved rather than quietly closed.

Task shape was left marked contested because your data and a third stack disagreed. That is now being resolved with a single-stack curve, see #11.

Thanks for running the failure analysis on your own first pass in public. That was the most useful part of the issue.