Laguna: thinking may be regime-dependent, not simply net-negative (needs review + independent test)

[TheTom]

This contests the guide's headline recommendation, so it is deliberately not merged. Requesting review and, ideally, an independent replication first. @Blackwellboy @Defilan, you both have stacks that could test this.

The claim

@apollo-mg measured thinking on HumanEval+, 492 samples, and got the opposite sign to our behavioral battery:

Regime	Winner	Evidence
Single-turn code generation, verifiable answer	thinking ON	90.85% vs 88.21% (+2.64 pts), flaky problems halve (11 vs 24)
Held-out behavioral / integrity	thinking OFF	ours: 94.2% vs 91.3%; ON fabricates bugs in clean code, over-refuses an authorized pentest, capitulates to a planted false premise
Long agentic loops	thinking OFF	ours: OFF 30/30 vs ON hung at 11/30; and false eliminates truncated-think (0/15 cap-hits vs 2/15)

If it holds, this guide has been costing readers about 2.6 points and double the flakiness on single-turn codegen by recommending thinking off across the board, and we never measured that price. That is why I would rather over-review it than ship it fast.

Two methodology findings come with it, and one of them indicts this repo:

• Cap-hits are failures, not truncations. p95 output 10,152 tokens, median 1,945, so ~12k is a better ceiling than 16k. Raising the budget does not convert cap-hitters into passes, because compression-ratio analysis says most are degeneration loops. Our own arms retried to 16k and 64k on exactly the assumption this refutes (see the §2 confound note).
• A loop-detection stopping rule is claimed as the highest-value harness change: 11/492 produced no extractable answer, ~87% of their gap to a larger model.

Why this is not merged

It is the weakest-sourced item in the current set. It reached me relayed as a screenshot: no published raw data, no logs, no harness, and I have not seen a writeup under the author's own name. Compare @Blackwellboy's lab repo and @Defilan's merged harness, both of which let me check the work. I am crediting apollo-mg on the strength of being a known contributor to a sibling repo, not on the strength of published data, and that distinction should be visible rather than smoothed over.

What would get this merged

Any one of these:

1. A replication on another stack. The single-turn-codegen arm is cheap: fixed max_tokens, enable_thinking true vs false, a verifiable-answer benchmark, and see whether ON beats OFF. The merged scripts/thinking-probes/thinking_ab.py already does the interleaving, nonce, and control-cell handling.
2. Raw data from @apollo-mg: per-sample results, the two sampling points, and the compression-ratio analysis behind "cap-hitters are degeneration loops."
3. A scope-limited merge, if replication is not available: land it explicitly as single-benchmark, single-turn, HumanEval+-only, with the sampling caveats attached, and do not change Quickstart Laguna S 2.1: make the guide answer "what should I do", + @BlackwellBoy's 12h soak #1.

Specific things I would like challenged

• Is the regime split real, or is it the benchmark? HumanEval+ is single-turn codegen with a verifiable answer. Our battery is behavioral and multi-turn. Those may be measuring genuinely different things, which is the charitable reading, or the split may be an artifact of what each of us chose to score.
• Does the +2.64 survive the sampling confound? Their thinking-on leg ran t0.7 and thinking-off t0.6. That is two variables, the same class of mistake I disclosed in our own arms.
• The task-shape contradiction is still unresolved. @Defilan's re-run had coding-shaped prompts reasoning least (2/6 vs 6/6 factual); this stack reports coding-shaped reasoning most. My guess, flagged as inference: an all-coding benchmark never varied shape, so it measures apparatus rather than shape. Would like that confirmed or shot down.

check_guides.py passes. Main currently reverts to the reviewed wording with a pointer here.

[Blackwellboy]

Taking the two questions you flagged for challenge, in the order I can actually help with.

1. Task shape: our grid varied shape and apparatus separately, and it supports your inference

You wrote that an all-coding benchmark never varied shape, so it measures apparatus rather than shape, and asked for that to be confirmed or shot down. Our C0 to C9 grid crossed four task shapes against ten apparatus levels, 40 samples per condition, so it can separate them.

Shape is real and it is large. Firing rate across the whole grid, all conditions pooled:

task shape	fired
math	92/100
code	62/100
reasoning	47/100
summarization	0/100

Summarization never fired once in any condition, including with no system prompt at all. That is a floor no amount of apparatus explains.

But code specifically is a conjunction, not a shape effect:

condition	code	math
C0 bare	10/10	10/10
C4 bare named persona	0/10	10/10
C7 full agent prompt	10/10	10/10
C8 agent prompt + tools	10/10	10/10

Code fires fully with no apparatus, collapses to zero under a bare named persona, and comes back to 10/10 once the full agent prompt is present. So "coding-shaped tasks suppress" and "coding-shaped tasks reason most" can both be true readings of the same model depending on which single apparatus level you happened to hold fixed.

Which is your inference, confirmed: a benchmark that holds shape constant at coding and carries no apparatus is sitting at our C0 cell, where code fires 10/10. It is measuring the apparatus axis at one point, not the shape axis. It cannot see the C4 collapse and it cannot see the summarization floor.

Raw: gate-study/logs/grid_turns.jsonl.

2. On the sampling confound in the +2.64

You already caught that thinking-on ran t0.7 and thinking-off t0.6. I would add that the direction of that confound is not obviously benign for the claim. On a verifiable-answer codegen benchmark the lower-temperature arm is usually the one you would expect to score higher, so a lower-temperature arm losing by 2.64 points is not the shape you would predict from temperature alone. That argues the effect is probably real rather than manufactured, but it also means the magnitude is not clean, and +2.64 is small enough that a two-variable comparison cannot really defend it as a number. Scope-limited merge with the magnitude marked provisional seems right to me, if replication does not arrive.

3. On cap-hits as failures rather than truncations

This one I can support from a different angle, and it generalizes past Laguna.

We ran a six-requirement acceptance-criteria coding task on Qwen 3.6 35B-A3B at a fixed 4096 ceiling, thirty times across three prompt conditions. It returned completely empty content, having spent the entire budget inside the reasoning block, in 28 of 30 runs. Laguna on the same task and ceiling did it 9 times in 30. Not wrong answers, no answer at all.

That is the same failure your §5f describes, on a different model and a different serving stack, which I think promotes "treat cap-hits as failures" from a Laguna harness note to a general benchmarking rule. It also matches the degeneration-loop reading rather than the needs-more-budget reading: 4096 is not a stingy ceiling for that task, and more of it was not going to help.

Raw: cross-model/logs/criteria_turns.jsonl.

4. On the regime split being real versus being the benchmark

I do not think our data settles this and I would not claim it does. What I can say is that the two regimes differ on more than task shape: single-turn codegen with a verifiable answer has no apparatus and no accumulated context, and your behavioral and long-agentic arms have both. Since apparatus is now an established axis with a large effect, "regime" and "apparatus dose" are currently confounded with each other in the comparison. Splitting them would need the codegen benchmark re-run with a full agent prompt attached, which is a different experiment from the one anybody has run.

Worth stating in the section if it lands, because otherwise the regime table reads as a property of task type when it may partly be a property of how much scaffolding each regime happens to carry.

5. Two small things

Nothing to do with the claim, just noticed while reading the diff.

• The card's footer still says three independent stacks while the validation row above it now says four.
• Quickstart row 1 and the §5f table are consistent, but Quickstart row 4's ~12k ceiling and §5g's advice on truncated-think interact: a 12k cap makes truncated-think turns more likely than a 16k one did, so the row 4 change slightly raises exposure to the §5g failure unless the placeholder-substitution fix is in place. Might be worth one clause pointing row 4 at §5g.

On replication

It is running now, on full-precision Laguna S 2.1 NVFP4 at rev 0761412, vLLM. HumanEval+, both arms, with the two things your review asked for built in:

• Temperature held identical across arms. That is the whole point of running it: the +2.64 currently rests on a two-variable comparison, and this arm removes the sampling half so what is left is the enable_thinking half on its own.
• A 12k ceiling, which also puts @apollo-mg's own p95-derived recommendation under test rather than just adopting it.
• Per-sample cap-hit and extractable-answer logging, so the degeneration-loop claim can be checked directly rather than inferred: if cap-hitters really are loops rather than truncations, that shows up per sample, and the 11-of-492 no-extractable-answer figure gets an independent comparison point.

Results to this thread when they land, expected within a day, either way. If it comes back with ON ahead on our stack too, that is the replication you asked for. If it comes back flat or reversed once temperature is controlled, that is worth knowing before this changes Quickstart #1, and I will say so just as plainly. Raw data published the way everything else in our lab repo is.

[Blackwellboy]

Correcting myself on one of the two small things above, before you spend time on it.

I said the card's footer still says three independent stacks while the row above says four. That is wrong, and it is my misreading rather than a bug in your diff. The two numbers count different sets: the header counts every stack including your own, the footer counts the ones independent of you. Main is "three stacks" in the header against "2 independent stacks" in the footer, and this PR moves both together to four and 3. Same convention, correctly updated. Nothing to fix.

The second bullet still stands as far as I can tell: a 12k ceiling makes truncated-think turns more likely than 16k did, so Quickstart row 4 and the §5g failure interact, and a pointer between them would be worth one clause.

Apologies for the noise on the first one.

[TheTom]

will fix the two small things thanks @Blackwellboy

[TheTom]

Thanks, this is the most useful review comment the repo has had. Taking your points in order, then what I am running.

1. Task shape: inference confirmed, and I had just talked myself out of it

Your pooled table is the thing I did not have. I re-derived it from grid_turns.jsonl before writing anything, and it matches exactly:

task shape	fired
math	92/100
code	62/100
reasoning	47/100
summarization	0/100

Awkward timing on my end and worth stating plainly: about an hour before your comment landed I shipped a commit that declared this inference wrong and retired it, on the strength of your C0-vs-C4 code split alone. That was an over-correction, it stood for exactly one commit, and it is fixed in #12 with the retraction visible rather than quietly reverted. The two parts do not compete: shape is a large independent effect (the summarization floor is a floor no apparatus explains) and code specifically is a persona conjunction. A shape-constant no-apparatus benchmark sits at C0 where code fires 10/10, so it cannot see the C4 collapse or the summarization floor. That is your confirmation, folded.

Everything from this comment that touches §2 is in #12, not here, since #12 is the section rewrite. Your review on that one is welcome, particularly on whether "task and persona interact" is more weight than one grid should carry.

2. The +2.64 and the direction of the confound

Your framing is better than mine and I have adopted it: on a verifiable-answer codegen benchmark the lower-temperature arm is normally the one you would expect to score higher, so a lower-temperature arm losing by 2.64 is not the shape temperature alone predicts. That argues the effect is real while the magnitude stays unclean. Agreed on scope-limited merge with the magnitude marked provisional if replication does not land.

3. Cap-hits as failures: promoted to a cross-model pattern

I verified both lanes per-sample from your published logs rather than from your table: cross-model/logs/criteria_turns.jsonl gives 28/30 empty-content and gate-study/logs/criteria_turns.jsonl gives 9/30. That is now a standalone entry in patterns.md on main, "An empty response at a token cap is a failure, not a truncation," with the two harness rules (score cap-hits as failures; log the rate per arm, because dropping them inflates whichever arm degenerates most, normally the thinking-on one). Cited to your runs.

4. Regime and apparatus are confounded

This is the sharpest point in your comment and I had not seen it. Single-turn codegen with a verifiable answer carries no apparatus and no accumulated context; our behavioral and long-agentic arms carry both. Since apparatus is now an established large-effect axis, "regime" and "apparatus dose" are confounded with each other in the very comparison this PR is about, so the regime table may partly be measuring scaffolding rather than task type. It will be stated in the section if this lands. Separating them needs the codegen benchmark re-run with a full agent prompt attached, which nobody has run, and I am noting it as the follow-up experiment rather than pretending this PR settles it.

5. The two small things

Noted on the footer, no apologies needed, and your own correction is right about the convention. Unrelated: main's footer now reads "2 independent replications" instead of "2 independent stacks", same convention, just less ambiguous.

The second bullet is fixed, in b199529 on this branch. You had the section letter off by one (§5f is truncated-think, §5g is the gfx1151 serving config) but the interaction you identified is real and I had the two recommendations sitting in the same row with no pointer between them. The advice is now split by regime, because 12k is a p95 of single-turn outputs:

• Single-turn / benchmark: ~12k, score cap-hits as failures.
• Agent loop: keep headroom, and separately make the loop survive a capped turn rather than relying on the cap never being hit, since the failure is silent and kills every subsequent turn.

What I am running now, on a third stack

Since you are covering NVFP4/vLLM, I have put ours on the other side of the serving axis so the two runs are not the same experiment twice: Q4_K_M GGUF on poolside's own llama.cpp fork at 04b2b72cb, GB10, -fa on --jinja --no-mmap, 4 slots. HumanEval+ 164 problems, both arms, scored by the official evalplus scorer rather than anything I wrote.

Controls, matching what your review and mine both asked for:

• temperature identical across arms (0.6/0.95 both), so the sampling half is removed and what is left is the enable_thinking half
• max_tokens identical at 12,288, which puts @apollo-mg's own p95-derived ceiling under test rather than adopting it
• arms interleaved per problem, not blocked
• cache_prompt: false plus a per-request nonce
• no system message at all, deliberately, to keep off the persona axis your grid just established as large
• per-sample cap-hit, extractable-answer, and reasoning unique-line-ratio logging, so degeneration-loop-vs-truncation is checkable rather than inferred

Then a follow-up pass re-running only the cap-hitters at a raised ceiling, which is the direct test of "raising the budget does not convert cap-hitters into passes." If they convert, that claim is wrong on this stack and I will say so.

Results and raw JSONL to this thread either way, including if it comes back flat or reversed. Two independent replications on two different quants and two different serving stacks is a much better basis for changing Quickstart #1 than one screenshot was.

Note for whenever this does merge: this branch predates today's §2 rewrite and the patterns additions, so it needs a rebase on main first. Its card still carries the retracted dose-curve row.

[Blackwellboy]

Glad the pooled table was the missing piece, and respect for leaving the over-correction visible instead of quietly reverting it. That is the standard this repo keeps setting.

Your llama.cpp run design looks right to me, and the deliberate no-system-message call is exactly the control that matters now. Coordination point so our two runs stay maximally comparable: mine is also no system message (deliberately the C0 cell), temperature fixed identically across arms, 12,288 ceiling, arms interleaved with nonces, all 164 HumanEval+ problems, evalplus scoring, per-sample cap-hit and extractable-answer logging. So the only intended differences between our runs are quant and serving stack, NVFP4/vLLM against Q4_K_M/llama.cpp, which is what makes the pair informative. If yours and mine land the same sign on opposite stacks, that is about as settled as this question gets short of Poolside's own data. Results to this thread as soon as my run completes, raw JSONL alongside as usual.

Two small notes on the cap-hit pattern entry, since I re-derived it from my raw before saying anything.

The numbers are all clean: 28/30 and 9/30 both reproduce, both really are three prompt conditions per model, and every one of those empty responses really did stop at exactly 4,096 completion tokens. Nothing to fix there.

One sentence overreaches slightly though. The entry says "the same request under a lower apparatus dose completes fine, which is what separates a loop from a genuine truncation." That holds for Laguna and not for Qwen. Per condition:

condition	Laguna completed	Qwen completed
C0 bare	9/10	0/10
C4 persona	9/10	0/10
C7 agent prompt	3/10	2/10

Laguna behaves the way the sentence describes, fine at low dose and collapsing under the agent prompt. Qwen returns nothing at every dose including the bare prompt. If anything that strengthens the entry's conclusion rather than weakening it, because a failure that does not depend on apparatus at all is even harder to read as a truncation, but the stated discriminator is Laguna-only and I would not want it carried as a general rule. Raw: gate-study/logs/criteria_turns.jsonl and cross-model/logs/criteria_turns.jsonl.

Also minor, you described that entry as being on main. It is currently in #12 rather than main, so it will land when that merges. Flagging only so it does not get lost if #12 gets reshaped.

[TheTom]

Interim from our replication, 77/328 requests in

Third stack, other side of the serving axis from @Blackwellboy's NVFP4/vLLM run: Laguna S 2.1 Q4_K_M GGUF on poolside's own llama.cpp fork at 04b2b72cb, GB10, -fa on --jinja --no-mmap, 4 slots. HumanEval+ 164 problems x 2 arms.

No pass-rate numbers in this comment. Scoring is done by the official evalplus.evaluate at the end, not incrementally, and I am not going to eyeball a partial pass@1 and then have to walk it back. What follows is only what the run measures directly.

Paired tasks with both arms complete (n=38):

arm	fired	cap-hits	no extractable answer	median out-tok	p95 out-tok	max out-tok	mean latency
ON	34/38	0	0	1,069	3,093	5,025	147s
OFF	0/38	0	0	138	462	558	39s

Four things worth putting on the record now, because two of them bear on the methodology half of this PR rather than on the +2.64.

1. enable_thinking: false is structurally off on a third stack. 0 of 38 fired, 0 of 61 counting unpaired rows. That is an independent confirmation of @Defilan's render-check finding on a different quant and a different fork, and it makes this the strongest-evidenced claim in the guide. It also means the arms here really are separated on the variable of interest.

2. Zero cap-hits at the 12,288 ceiling, so far. Not one request in 77 has hit it. If that holds to 328, the cap-retry pass I built has nothing to re-run, and the honest reading is not "the claim is wrong" but "the ceiling is not the binding constraint on this stack", which is a different and weaker statement than the guide currently makes.

3. Our p95 is roughly a third of theirs, which is the interesting divergence. @apollo-mg reported p95 10,152 and median 1,945, and derived ~12k from it. We are measuring p95 3,093, median 1,069, longest single response 5,025. Same benchmark, same task shape, no system message in either case. So the ~12k figure looks like a property of their stack and sampling rather than of the model, and a guide-level "use ~12k" recommendation sourced from one stack's p95 is doing more extrapolating than the data supports. Worth reconciling against @Blackwellboy's run when it lands, since he is also using 12k.

4. The cost gap is large and has to be part of the verdict. ON is spending 7.7x the output tokens (1,069 vs 138 median) and 3.8x the wall clock (147s vs 39s). Whatever pass-rate delta comes out, that is the price it has to beat, and a +2.6 point gain bought at 7.7x tokens is a different recommendation from a +2.6 point gain that is free. The current PR text does not price it at all, and it should.

Incidental cross-check on @Blackwellboy's grid, free from this run: 164 bare code-shaped prompts with no system message and thinking on is the same cell as his C0 code, which fired 10/10. We are at 34/38 (89%) and the four that did not fire are HumanEval/3, /8, /9, /15. So code firing is high but not saturated here, in the same direction as the bare-prompt gap already noted in the guide (his 75% against our ~50%).

Timeline: 389 request-minutes of work left at 4 concurrent slots, so ~1.6 hours to all 328 rows, then official evalplus scoring on both arms plus the cap-retry pass if anything hit the ceiling. Numbers and raw JSONL to this thread after that, including if it comes back flat or reversed.

I am deliberately not raising parallelism to speed this up: the model is 71 GB resident on a 121 GB unified-memory box, and doubling the KV allocation to get more slots is how that box starts swapping and stops being a valid measurement.

[apollo-mg]

Figured I'd jump in with what I've got so far.

Laguna-S-2.1 test data — Apollo fleet

Raw per-sample data behind the numbers quoted in #10.

Hardware (all runs): quad Tesla P100-PCIE-16GB (sm_60), 1063 MHz / 150 W, single node.
Server: poolside llama.cpp fork, -c 32768 -np 1 -ngl 99 -sm layer -ts 1,1,1,1 -fit off
Model: Laguna-S-2.1-UD-Q2_K_XL.gguf — Q2_K_XL. This matters; see CONFOUNDS.
Benchmark: HumanEval+ (EvalPlus extended tests), full 164 problems.
Dataset identity is fingerprint-verifiable with humaneval-plus/fetch_dataset.py
(164 problems, 10,730,825 test bytes, HumanEval/0 test = 77,039 chars).
max_tokens: 16000 on every run.

---

1. Thinking ON vs OFF — HumanEval+ full 164, K=3 (492 samples per arm)

Each arm at its own card-recommended sampling.

	thinking ON	thinking OFF
sampling	t0.7 / top_p 0.95 / top_k 20	t0.6 / top_p 0.95 / top_k 20
pass@1 pooled	90.85 % (447/492)	88.21 % (434/492)
per-sweep	90.9 / 90.2 / 91.5 %	86.6 / 90.2 / 87.8 %
PASS	447	434
WRONG	30	56
TRUNCATED (hit 16k cap)	10	0
NO_ANSWER	1	0
EXEC_TIMEOUT	4	2
solved 3/3	143 / 164	131 / 164
never solved	10 / 164	9 / 164
flaky (1–2 of 3)	11 / 164	24 / 164
median out_toks	1 945	274
p95 out_toks	10 152	734
max out_toks	16 000	1 079
wall clock	72 525 s	9 119 s

Files: humaneval-plus/laguna_hep_results_t07_k3.json (ON),
laguna_hep_results_t06_k3_nothink.json (OFF). Per-problem buckets, finishes,
srcs, out_toks, rc_chars.

Thinking-ON is inferred, not directly measured, for the t0.7 arm. That run predates the
harness's enable_thinking plumbing, so its metadata carries no thinking flag and no
reasoning_content lengths. Supporting evidence: token medians ON > OFF on 164/164
problems, median ratio 6.51×, 0/492 ON samples at or below the OFF median.

2. Stopping-rule observation

WRONG counts are near-identical between Laguna-ON (30) and a much larger model run on the
same node, same harness, same 492-sample design (Puzzle-75B-A9B-UD-IQ4-XL, 28 WRONG,
93.90 % pass@1 — humaneval-plus/puzzle_hep_results_t10_k3.json).

Laguna produced no extractable code on 11/492 samples; Puzzle on 1/492.

Conditioning on samples that produced code, the 3.05-point gap narrows to 1.16 pt
(EXEC_TIMEOUT counted as wrong code) or 0.58 pt (counted as non-answer). Both
definitions give the same direction.

Counter-evidence against over-reading it: forcing termination via enable_thinking: false
removed all 11 non-answers but moved WRONG 30 → 56, a net loss of 2.64 points.

3. Persona × tools factorial

15 HumanEval+ problems (every 10th), K=1, t0.7 / top_p 0.95 / top_k 20, thinking at
template default. Each cell in its own subshell so env could not leak.

Persona string: "You are a senior software engineer. Write clean, correct, production-quality Python."
Tools: 3 schemas (read_file, write_file, run_command) on every request, never called back —
no tool results re-entered context.

cell	system prompt	tools	thinking fired	mean reasoning	median per-problem ratio vs base	pass@1
base	default	no	15/15	8 418 ch	—	15/15
persona	PERSONA	no	15/15	12 704 ch	1.09×	13/15
tools	default	yes	15/15	6 377 ch	0.92×	15/15
both	PERSONA	yes	13/15	3 330 ch	0.39×	13/15

Independent composition would predict 1.09 × 0.92 ≈ 1.00×; observed 0.39×.

Robustness (n=15, so this matters):

comparison	sign test	median ratio	drop largest single gap
base vs persona	9/15 longer — coin flip	1.09×	0.96×
base vs tools	11/15 shorter	0.92×	0.73×
base vs both	14/15 shorter	0.39×	0.37×

The persona row's ratio of means is 1.51×, which reads as "+51 % reasoning." That is one
problem: HumanEval/30, 2 260 → 71 375 chars. Drop it and the ratio is 0.96×. The mean
was reported first and had to be retracted.

Files: persona-x-tools/sup_results_{base,persona,tools,both}.json,
analysis in cell_pairwise.py.

---

CONFOUNDS AND SCOPE LIMITS

1. Q2_K_XL only. The 2-bit tax is entangled with every result here, and specifically
   with the wedging. A Q4_K_M run reporting a much lower token ceiling and zero cap-hits
   would be consistent with the wedging being a quant artifact rather than a Laguna
   property. This has not been tested here.
2. "Cap-hitters are degeneration loops" is not established by this data. That claim
   comes from an earlier temp-0 K=1 leg via compression-ratio stratification (loops
   < 0.08, coherent-capped > 0.12). An attempt to validate that threshold on a second model
   failed to test it — the model never wedged, so there were no positives. Treat as
   inferred.
3. Single benchmark, single turn. HumanEval+ is single-turn code generation. No
   long-agentic-loop data here at all.
4. The 2×2 is K=1 over 15 problems. It sizes an effect; it does not establish a rate.
   Firing failed only in the both cell, 2/15 — Fisher p ≈ 0.48 against 0/15, i.e. not
   statistically distinguishable. The defensible claim is the length collapse, not a
   firing-rate claim.
5. The persona bundles an identity claim with a quality demand. Which half drives the
   interaction is unseparated.
6. Tools were passed but never called back. A real agent loop feeds tool results into
   context every turn; that is a further untested and plausibly stronger variable.
7. Server flags for the two eval legs were not recorded (they predate the launcher script).
   -fa on/off was never tested on Laguna.
8. The 15 problems in the 2×2 are every 10th HumanEval+ id, not a random sample.

Files

humaneval-plus/
  laguna_hep_results_t07_k3.json      thinking ON, per-problem
  laguna_hep_results_t06_k3_nothink.json  thinking OFF, per-problem
  puzzle_hep_results_t10_k3.json      larger-model comparison, same harness/node
  *.log                               console output for each
  hep_eval.py                         harness (revision as of 2026-07-25)
  fetch_dataset.py                    rebuilds + fingerprint-verifies the dataset
  SUMMARY.md                          full writeup incl. method notes
persona-x-tools/
  sup_results_{base,persona,tools,both}.json
  sup_{base,persona,tools,both}.log
  cell_pairwise.py                    paired/sign-test analysis
  SUMMARY.md                          full writeup

laguna_apollo_data_20260726.tar.gz

[Blackwellboy]

This is exactly what the thread needed, thanks for shipping the raw. The Q2_K_XL disclosure is the big one: it means the three replications now span the quant axis by accident, yours at 2 bit, Tom's at Q4, mine at NVFP4, all on HumanEval+ both arms. If the sign survives all three, that's a model property. If it fades with quant quality, that's its own finding. Mine also logs thinking firing directly per sample (the plumbing yours predates), so we'll get a measured rather than inferred ON arm at the high end. Results shortly, the run is nearly complete.

One note from adjacent data: your caution on cap-hitters as degeneration is warranted. On Qwen the identical symptom turned out to be pure truncation, budget converts 8 of 10 empties to valid at 8192 with zero degeneration in any tail. Different model, but it says the symptom has more than one cause and per-model checking is right.

[apollo-mg]

Shared your response with my agent, got a test queued up.

Opus 5: HumanEval/47 hit the 16 k ceiling twice and passed both times — extractable, correct code, then it kept generating until the cap. So finish_reason=length isn't a failure signal on its own, and my config-doc line saying cap-hits must be bucketed as failures was wrong. Fixed, with the correction visible rather than reworded away.

The full breakdown of the 12 cap-hits:

│ cap-hit │ bucket │ problem pass_frac │
│ HumanEval/47 ×2 │ PASS │ 1.00 │
│ /44, /90 │ TRUNCATED │ 0.67 │
│ /118 ×2 │ TRUNCATED │ 0.33 │
│ /76, /116 ×2, /132, /145 ×2 │ TRUNCATED │ 0.00 │

6 of the 10 truncations sit on problems Laguna never solves in any sample — which is consistent with "budget won't help" but doesn't demonstrate it, because a never-solved problem is also never solved with more budget. Both hypotheses predict the same thing there. The 4 truncations on solvable problems — /44, /90, /118×2 — are the only place the hypotheses diverge, and that's the population BlackwellBoy's test has to target.

The decisive experiment is now specified and ready: re-run those 8 problems at a much larger budget and watch whether the 4 solvable ones convert. One wrinkle worth knowing — our 16 k ceiling wasn't a policy choice, it's the server's -c 32768, so testing this needs Laguna relaunched with a bigger context, not just a bigger max_tokens. If truncations convert, our stopping-rule recommendation is wrong for this model and the fix is simply more budget. If they cap again at 48 k, degeneration is confirmed.

It's queued behind stage 2 on .194, same as the loop detector — and it's the better experiment of the two, since it has guaranteed positives where the detector test didn't.

Worth noting what BlackwellBoy set up by accident: the three replications now span 2-bit, Q4_K_M, and NVFP4 on the same benchmark and both arms. If our +2.64 survives at NVFP4 with directly-measured firing, it's a model property; if it fades as quant quality rises, that's a cleaner finding than the original. Either outcome is worth more than our single point, and his run gets the measured ON arm that ours could only infer.

[Blackwellboy]

Replication results, as promised. Full-precision Laguna S 2.1 NVFP4, rev 0761412, vLLM 0.25.1, HumanEval+ all 164 problems, both arms, temperature identical.

Verdict first

The +2.64 does not replicate once the temperature confound is removed. On HumanEval+ — the metric the claim was made on — thinking ON scored 89.84% vs OFF 90.85% (mean of 3 seeds; ON −1.02). Paired per problem it is simply flat: 10 problems favor ON, 13 favor OFF, 141 tied. On base HumanEval the delta leans the other way (ON +1.22). Both deltas are inside seed noise, and both are smaller than the gap between scoring variants of the same benchmark within one run.

So on a second stack, with one variable instead of two, the accuracy half of the single-turn-codegen row is flat-to-reversed, not +2.6. The temperature difference appears to have been the whole accuracy story — consistent with your own review note that a lower-temperature arm losing by 2.64 was not the shape temperature alone would predict.

What DOES replicate — both of the methodology claims

1. Flakiness: direction confirmed. ON 11 flaky problems vs OFF 17 (theirs: 11 vs 24). Not halved here, but ON is genuinely more run-to-run stable. This is the defensible part of the regime row.
2. Cap-hits are failures, not truncations: strongly confirmed. 15/492 ON runs hit the 12,288 ceiling; 14 of them returned zero extractable code, whole budget inside the think block, tail compression ratios 2.9–143× (coherent text ~2.5–3×; 44/133/143 are hard loops). ON's p95 completion is 6,763 tokens — cap-hitters are not the tail of the length distribution, they are a separate failure mode more budget cannot fix. The loops concentrate on specific problems (HumanEval/116 and /132 looped in all 3 ON seeds). One OFF run also looped (40k chars of content, ratio 20) — same failure mode, 1/15th the rate.
3. ~12k ceiling: adequate on this stack. Nothing non-looping came near it; 12k vs 16k loses nothing and caps the cost of each loop.
4. No-extractable-answer: ON 22/492 (~4.5%: the 14 loops + 8 finish=stop responses with no fenced code) vs OFF 1/492. Their 11/492 is the right order of magnitude for ON. A loop-detection stopping rule would recover at most ~1 pt for ON here — i.e. it brings ON to parity on HumanEval+, not ahead.

Design (the confound eliminations your review asked for)

• Temperature identical both arms: t 0.7, top_p 0.95, top_k 20 (model-card sampling), sent explicitly per request. 3 seeds per (problem, arm), same seed both arms of a pair; pass@1 = mean ± sd.
• No system prompt, user message only — deliberately the C0 cell from our grid, which is where an all-coding no-apparatus benchmark sits. With the kwarg on, thinking fired 492/492 (C0 code = 10/10 reproduces); control arm fired 0/492.
• Arms explicit enable_thinking: true|false; byte-identical HumanEval+ prompts (no nonce — per-request vLLM seeds instead; vLLM prefix caching is exact-KV reuse, so the llama.cpp cached-reply hazard the nonce guards against doesn't exist here); arms interleaved per problem; max_tokens 12,288 fixed, no retry; scoring = evalplus 0.3.1 sanitize + its own test execution, no LLM in the scoring path; 984 requests, 0 errors.

Numbers

arm	HumanEval base	HumanEval+	flaky	cap-hit	no-extract	mean wall/problem
ON	95.73 ± 1.06	89.84 ± 0.35	11	15/492	22/492	200.2 s
OFF	94.51 ± 0.61	90.85 ± 1.61	17	1/492	1/492	18.5 s

That last column is worth a sentence: on this stack thinking ON costs ~11× wall clock for a flat accuracy result. Whatever the regime table ends up saying, "ON for single-turn codegen" is not free.

What I'd suggest for the merge decision

• Land §5f's methodology findings (cap-hits-are-failures, ~12k ceiling, loop-detection rule) — two-stack confirmed now, and the loop-degeneration mechanism generalizes (we've seen the same signature on Qwen 3.6 at a 4096 ceiling, 28/30).
• Do not change Quickstart Laguna S 2.1: make the guide answer "what should I do", + @BlackwellBoy's 12h soak #1 to "true for single-turn codegen." The accuracy half of that row didn't survive temperature control on a second stack. If the row lands at all, scope it to stability: "ON reduces run-to-run flakiness on verifiable codegen (two stacks), accuracy effect not reproducible, costs ~11× latency."
• The scope-limited-merge option in your list fits the data better than the full regime table.

Raw data: per-sample JSONL (984 rows: full content + reasoning, cap-hit, extractable, compression ratios, seeds, wall times), driver, and analysis script — pr10-replication/ in the lab repo, published the same way as everything else there. The full-precision spine baseline that closes our quantization question is alongside at spine-fullprecision/.

One serving note for §2, found while running this: the stray leading </think> we and @Defilan have both seen is not a container misconfig — it appears whenever the kwarg is absent and the model skips thinking (poolside_v1 parser on vLLM leaves the orphan close-tag), and never appears when the kwarg is explicit either way (0/984 rows here). One more reason for Quickstart #1's "set it explicitly," whatever value it ends up recommending.

[Blackwellboy]

Respect for correcting that in public with the mistake visible, that's the standard this thread keeps setting.

Your /47 case is a good catch and it's exactly why my run bucketed on zero extractable code rather than on finish_reason. A cap-hit that still produced correct code is not a failure and should not be scored as one. I had the same case and it went the same way: HumanEval/32 seed 2 hit my 12,288 ceiling with extractable code, tail compression 2.88 which is the lowest ratio in my whole cap population, and it was not scored as a failure. That is 1 of my 15 cap-hits, and the other 14 returned zero code. The two criteria are genuinely not interchangeable on my data either: 22 of my ON runs had no extractable code while only 15 hit the ceiling, so bucketing on finish_reason would have both over-counted and under-counted different failures.

On the discriminating question, my NVFP4 run has partial data for you already. p95 output on the ON arm was 6,763 tokens against a 12,288 ceiling, so most of that arm was nowhere near budget limited, and 14 of my 15 cap-hits returned zero code with the entire budget consumed inside the think block, tail compression 2.88 to 142.86. The looping concentrated on specific problems rather than spreading: HumanEval/116 and HumanEval/132 capped in all three seeds, accounting for 6 of my 15 cap-hits across 10 distinct problems. That reads as degeneration rather than starvation at this quant.

The more useful part for your queued experiment: your three discriminating problems already have NVFP4 data.

problem	your 2-bit result	my NVFP4 ON arm (3 seeds)
/44	truncated, pass_frac 0.67	0 cap-hits, extractable 3/3
/90	truncated, pass_frac 0.67	0 cap-hits, extractable 3/3
/118	truncated ×2, pass_frac 0.33	1 cap-hit of 3, extractable 2/3

So two of the three never approach the ceiling at NVFP4 and the third caps once in three. I would not read that as settling your question, because quant and ceiling both differ between us, but it is consistent with quant quality rather than budget being what moves those specific problems.

Worth noting the overlap the other way too: five problems cap on both of our stacks despite the 2-bit versus NVFP4 gap, /76, /116, /118, /132 and /145. The same problems degenerating at both ends of the quant axis is a stronger argument for problem-specific degeneration than either of us could make alone.

Your four solvable-problem truncations are still the cleanest test, and the 48k re-run will settle it better than anything I ran. Good catch on the ceiling really being the server's -c 32768; that is the kind of thing that quietly invalidates a re-run if you don't relaunch.

Full numbers and the 984-row raw are in my thread comment above and in the repo if useful for comparison.

[apollo-mg]

I'll probably need to grab another quant or two. While our test is running, here's my agent's assessment:

---

Opus 5: Two things worth having, one of which corrects my correction.

Our harness was already right; my prose was the error. Checked directly: finish=length fires 12 times, src=none (no extractable code) fires 11, and our TRUNCATED bucket requires src=='none' — which is exactly the criterion he's advocating. That's why /47 scored PASS. The bug was in what I wrote in the config doc, not in what the harness did.

And his NVFP4 numbers plus ours decompose the cap population in a way neither set does alone:

│ problem │ our Q2 (16 k ceiling) │ his NVFP4 (12,288 ceiling) │ reading │

│ /44 │ cap 1/3, ext 2/3, pf 0.67 │ 0 cap-hits, ext 3/3 │ quant-driven │

│ /90 │ cap 1/3, ext 2/3, pf 0.67 │ 0 cap-hits, ext 3/3 │ quant-driven │

│ /118 │ cap 2/3, ext 1/3, pf 0.33 │ 1 of 3, ext 2/3 │ intermediate │

│ /116 │ cap 2/3, ext 1/3, pf 0.00 │ caps all 3 seeds │ problem-intrinsic │

│ /132 │ cap 1/3, ext 2/3, pf 0.00 │ caps all 3 seeds │ problem-intrinsic │

The ceiling difference makes this stronger, not weaker. His ceiling is lower — 12,288 vs our 16,000 — so all else equal he should cap more often. That /44 and /90 cap zero times at NVFP4 despite the tighter budget is real evidence those two are fixed by quant quality.

Meanwhile /116 and /132 cap harder at full precision than at 2-bit (3/3 vs 2/3 and 3/3 vs 1/3). Those aren't quantization damage — they're something about the problems themselves that survives a total change of stack, engine, and precision.

So the five-problem overlap he identified isn't one phenomenon. It's at least two: quant-sensitive degeneration that better weights fix, and problem-intrinsic degeneration that nothing so far touches. That also sharpens our 48 k test — it's the only clean budget-only manipulation at fixed quant either of us can run, and it should be read per-problem rather than pooled, because those two classes will answer differently.

One discrepancy worth flagging before the "same order of magnitude" framing sets: his no-extractable-answer rate is 22/492 against our 11/492, but the composition differs sharply — his is 14 cap + 8 finish=stop with no fenced code, ours is 10 cap + 1. An 8× difference in the stop-without-code population is more likely an extraction-criterion difference (evalplus sanitize vs our extractor) than a model difference. If those rates get compared directly, that needs saying.

[TheTom]

Replication results: third stack, Q4_K_M on poolside's own fork

Run complete, 328/328 requests, 0 errors. Scored by the official evalplus 0.3.x scorer, not by anything I wrote.

Verdict first

The +2.64 does not replicate here either. It reverses. On HumanEval+, thinking ON scored 88.4% against OFF 90.9%, so ON is 2.5 points worse, and on base HumanEval it is also behind (94.5 vs 95.1). That is the opposite sign to the claim, on the same benchmark, at a third quantization, with temperature held identical.

Combined with @Blackwellboy's NVFP4 result, both runs that removed the temperature confound came back flat-to-negative, at two different temperatures and two different engines:

stack	quant	engine	temp ON/OFF	HumanEval+ ON	HumanEval+ OFF	ON delta
@apollo-mg	Q2_K_XL	poolside llama.cpp	0.7 / 0.6	90.85	88.21	+2.64
@Blackwellboy	NVFP4	vLLM 0.25.1	0.7 / 0.7	89.84	90.85	−1.02
ours	Q4_K_M	poolside llama.cpp 04b2b72cb	0.6 / 0.6	88.41	90.85	−2.44

The one arm that shows the gain is the one where the two arms ran at different temperatures. That is now two independent refutations rather than one, and our run is deliberately the awkward middle of the accidental quant axis @Blackwellboy noticed: same engine as Apollo, same hardware and same 12,288 ceiling as BlackwellBoy. So the divergence is not explained by engine or by ceiling.

Curiosity worth recording: all three runs land on 90.85% for one arm or the other. Apollo's ON, BlackwellBoy's OFF, and our OFF are the same number to two decimals. That is 149/164, and it is almost certainly a coincidence of a saturated benchmark rather than anything meaningful, but it is the kind of coincidence worth naming out loud before someone reads significance into it.

Paired, per problem

Pass-rate deltas on a saturated benchmark hide how few problems actually move:

• 148 of 164 tied
• 10 problems only OFF solved: /6, /7, /22, /47, /55, /86, /89, /134, /156, /162
• 6 problems only ON solved: /10, /49, /93, /97, /102, /124

Note that 2 of OFF's 10 wins (/47 and /156) are problems where ON burned its whole budget in the think block and returned no code at all. So a chunk of the accuracy gap is not "ON reasoned to a wrong answer," it is "ON never answered."

Cap-hits: the degeneration reading is confirmed, and cleanly

4 of 164 ON runs hit the 12,288 ceiling. All 4 returned zero extractable code, entire budget spent inside the think block. OFF: 0 cap-hits, 0 unextractable, 0 fired, all 164.

The degeneration signature separates on our data too, using unique-line ratio over the reasoning tail:

	median unique-line ratio
cap-hitters	0.320
non-cap-hitters	0.551

Per sample:

problem	reasoning chars	uniq-line ratio	extractable
HumanEval/47	36,651	0.185	no
HumanEval/132	47,025	0.448	no
HumanEval/145	44,340	0.385	no
HumanEval/156	36,733	0.256	no

And ON's p95 completion is 5,261 tokens against a 12,288 ceiling, so as on @Blackwellboy's stack the cap-hitters are not the tail of the length distribution, they are a separate failure mode.

Cross-stack problem overlap is the strongest part of this. Of our 4 loopers, 3 also loop on at least one other stack: /132 loops on all three of us, /145 on ours and Apollo's, /47 on ours and Apollo's. That is problem-specific degeneration surviving a 2-bit to NVFP4 quant spread, which none of us could have established alone.

On HumanEval/47 specifically, and this cuts against the counter-example. Apollo's /47 capped twice and passed both times with correct extractable code, which correctly prompted the retraction of "bucket all cap-hits as failures." On our stack /47 capped and returned zero code, at the lowest unique-line ratio in our whole population (0.185), and it is one of OFF's 10 paired wins. So /47 is not reliably benign; it is benign at 2-bit/16k and degenerate at Q4_K_M/12k. The right conclusion stays the one both of you converged on independently: bucket on zero-extractable-code, never on finish_reason. Our data agrees, just from the other direction.

The ceiling

Our p95 is 5,261 against Apollo's 10,152 and @Blackwellboy's 6,763. So ~12k is comfortable here and nothing non-looping came near it.

I want to correct my own interim comment on this thread: at n=38 I reported p95 3,093 and said ours was "roughly a third of theirs." At the full 164 it is 5,261, about half. The direction held but I quoted a number off a partial run and it moved, which is exactly the mistake this thread keeps catching in other people's work.

Cost, which the PR still does not price

	ON	OFF	ratio
total output tokens	359,027	40,450	8.9x
total wall clock	638 min	86 min	7.5x
median tokens/problem	1,564	192	8.1x

@Blackwellboy measured ~11x wall clock, we measure 7.5x. Thinking ON costs roughly an order of magnitude for an accuracy result that is flat at best and negative on two of three stacks.

Thinking control, third independent confirmation

• enable_thinking: false: 0/164 fired, 0 unextractable, 0 cap-hits. Structurally off, third stack, third serving path.
• enable_thinking: true: 156/164 fired (95%).

The 8 that did not fire: /3, /8, /9, /15, /48, /58, /85, /112. Relevant to the merged §2 work: this is the C0 bare-code cell, where @Blackwellboy measured code firing 10/10 and, in this replication, 492/492. We get 95%. Same cell, same absent system prompt, different serving path, consistently a bit lower, which matches the bare-prompt gap already recorded in the guide.

Design

• Temperature identical across arms, 0.6 / top_p 0.95 both, deliberately a different point from @Blackwellboy's 0.7 so the two controlled runs do not share a temperature.
• max_tokens 12,288 fixed, no retry, matching his ceiling.
• No system message at all, the C0 cell.
• Arms interleaved per problem; cache_prompt: false plus a per-request nonce (llama.cpp, so the cached-reply hazard is real here in a way it is not on vLLM).
• Firing read from reasoning_content or reasoning, per the fix in spine-probes: support thinking models in run_probes and judge #9.
• Unextractable responses scored as failures, not dropped.
• Serving: --jinja -fa on --no-mmap -ngl 99 -c 65536 -np 4, poolside fork 04b2b72cb, Q4_K_M, GB10.

Limitations, stated plainly

1. K=1. Both other runs are K=3 with 492 samples per arm; ours is 164, one seed. Our per-arm number carries more noise than theirs and we cannot report flakiness at all, which is the one axis where ON genuinely wins on both other stacks. Our −2.44 should be read as agreeing in sign with @Blackwellboy's −1.02, not as a precise magnitude.
2. Single benchmark, single turn. Says nothing about the long-agentic regime where the original disagreement lives.
3. One temperature point per arm, no sweep.
4. Prompt wrapper is ours, not either of theirs, so small extraction differences are possible. Extraction was clean: 160/164 ON and 164/164 OFF, and every failure was a cap-hit.

Still running

The cap-retry: the 4 cap-hitters re-issued at 32,768, everything else identical, which is the direct test of "more budget does not convert cap-hitters into passes." Per @apollo-mg's warning, our 12,288 ceiling was also effectively a server limit (-c 65536 -np 4 gives 16,384 per slot), so this needed a relaunch at -c 49152 -np 1 rather than just a bigger max_tokens. That catch saved the experiment from being invalid. Results to follow.

Raw JSONL (328 rows: firing, cap-hit, extractability, unique-line ratios, token counts, latencies, full solutions), runner, scorer, and cap-retry script available if useful for comparison.

What I now think should happen to this PR

• Do not change Quickstart Laguna S 2.1: make the guide answer "what should I do", + @BlackwellBoy's 12h soak #1. The accuracy half failed to replicate on two independent stacks, and reversed on ours.
• Land the methodology findings, now three-stack confirmed: cap-hits are failures, bucket on zero-extractable-code rather than finish_reason, ~12k is an adequate ceiling, and a loop-detection stopping rule is worth building.
• If the regime row lands at all, scope it to stability and price it: ON reduces run-to-run flakiness on verifiable codegen (two stacks, and we cannot speak to it at K=1), the accuracy effect does not reproduce, and it costs 7.5x to 11x wall clock.

That is @Blackwellboy's recommendation and our data independently supports it.

[TheTom]

Follow-up: what the thread as a whole now establishes, and what it does not

Separate comment on purpose. The one above is our data. This one is about how the four sets of results interact, including where other people's comments change the reading of our claims, not just Apollo's.

1. The strongest result in this thread is not about thinking. It is about method.

The original claim went through three stages: published as a screenshot with no data → replicated twice under a fixed confound → reversed. Nothing about that is a knock on @apollo-mg, who published the full raw with a confounds section that flagged the Q2_K_XL entanglement himself, before anyone asked. Worth being precise about what actually moved the number: not sample size, not hardware, not the benchmark. One variable, temperature, held constant.

Both controlled runs went negative. They used different temperatures (0.7/0.7 and 0.6/0.6), different engines, different quants, different ceilings, and different sample counts. About the only thing they share is that each arm ran at the same temperature as its partner. That is a cleaner result than a positive one would have been.

2. What upgraded our own claims

Our "thinking is net-negative" headline survives, but the reason I gave for it was too narrow. The guide justified it from held-out behavioral work and long agentic loops, and explicitly conceded single-turn codegen as the regime where ON might win. Three stacks now say ON does not win there either on accuracy. So the recommendation was right and the carve-out I was preparing to add was not needed. That is a case where the review process saved the guide from a change rather than forcing one.

enable_thinking: false is now the single best-evidenced claim in the guide. Four independent measurements: @Defilan's deterministic /apply-template render check, @Blackwellboy's 0/492, ours 0/164, and the structural mechanism (pre-closed </think>). Different quants, engines, and hardware. Nothing else in the guide has that.

@Blackwellboy's orphan-</think> finding closes a loose end I had been treating as a container misconfig. The stray leading close-tag appears only when the kwarg is absent and the model skips thinking, and never when the kwarg is explicit either way (0/984 rows). That converts "set it explicitly" from a correctness argument into a correctness and output-hygiene argument, and it explains a symptom both he and @Defilan had seen separately. Folding into §2.

3. What downgraded our own claims

Our p95 claim moved under me and I have corrected it on the thread. I reported "p95 is a third of theirs" from a partial run; at full n it is about half. Same class of error as quoting a table instead of re-deriving from raw, which is the thing I have been asking others to avoid.

Our K=1 is the weakest methodology in the thread. Both other replications are K=3 / 492 samples. We cannot report flakiness at all, and flakiness is the one axis where ON genuinely wins on both stacks that can measure it. So on the part of the regime claim most likely to survive, we have nothing to contribute, and our accuracy number should be read as a sign agreeing with @Blackwellboy's rather than as an independent magnitude.

The task-shape resolution merged in #12 also needs the prompt-identity caveat carried here. @Blackwellboy found that shape in his grid is confounded with prompt identity, one fixed prompt per task type. Our HumanEval+ run is the opposite trade: 164 genuinely distinct coding problems, but only one shape and one apparatus level. Neither design alone separates shape from prompt from apparatus. Together they bracket it: his varies apparatus with prompt fixed, ours varies prompt with apparatus fixed. Both point the same way on the C0 code cell (his 10/10 and 492/492, our 95%), which is more than either shows alone.

4. Where the thread still disagrees, honestly

Cap-hits: same symptom, more than one cause, and this is now well established. Three positions that all turn out to be right in their own scope:

• @apollo-mg's HumanEval/47 passed twice at the cap on Q2_K_XL, which retired "bucket all cap-hits as failures."
• Our /47 capped with zero code at the lowest unique-line ratio in our population on Q4_K_M. Same problem, opposite character.
• @Blackwellboy's adjacent Qwen data: budget converted 8 of 10 empties to valid at 8192 with no degeneration in any tail, i.e. genuine truncation.

The synthesis all three of us reached independently is the operational one: bucket on zero-extractable-code, never on finish_reason, and check per model. That is a better rule than any of us started with, and it came out of disagreement rather than agreement.

The regime-versus-apparatus confound is unresolved and nobody's data touches it. @Blackwellboy's point stands: single-turn codegen has no apparatus and no accumulated context, our behavioral and long-agentic arms have both. Since apparatus is now an established large-effect axis, "regime" and "apparatus dose" are confounded in the exact comparison this PR is about. Every run in this thread, all four, sits at the no-apparatus end. The discriminating experiment is the codegen benchmark re-run with a full agent prompt attached, and it has not been run by anyone. It is the highest-value open item here and I would rather name it than let the regime table imply it is settled.

5. What I am taking away about running this repo

Four people, four stacks, and the two claims that survived contact are the two that were mechanically checkable: a template rendering, and a per-sample bucket rule. The two that did not survive were both single-stack inferences from summary statistics: my dose curve and the +2.64. That is not a coincidence, and it suggests the bar for anything entering a guide should be "can someone re-derive this from published raw on their own hardware," not "is the number large."

Both of the guide's biggest corrections today came from someone re-deriving my numbers from raw and finding the reasoning wrong while the arithmetic was right. Worth saying plainly: this thread has been more useful to offlabel than my own testing was.

Cap-retry results to follow, and I will report them the same way whether or not they support the reading I have argued for above.

[TheTom]

Cap-retry results: neither hypothesis was right, and the reason matters

Promised follow-up. The 4 cap-hitters from our Q4_K_M run, re-issued at 32,768 (2.7x the original ceiling), everything else identical. Per @apollo-mg's warning this needed the server relaunched at -c 49152 -np 1, since our 12,288 was really a per-slot limit rather than a policy choice.

problem	old	new	capped again	extractable	uniq-line ratio (retry)
HumanEval/47	12,288	3,767	no	yes	0.600
HumanEval/132	12,288	6,985	no	yes	0.498
HumanEval/156	12,288	2,153	no	yes	0.606
HumanEval/145	12,288	32,768	yes	no	0.086

Read the middle column before the verdict column. Three of the four "converted", but they finished at 2,153, 3,767 and 6,985 tokens, all well below the original 12,288 ceiling they supposedly needed more budget than. More budget cannot be what fixed them. They simply did not loop on the new draw.

So the honest finding is a third thing neither side proposed: on this stack the loops are stochastic per sample, not deterministic per problem. Same problem, same prompt, same temperature, same everything except the draw: loops one time, finishes in 2k tokens the next. That reframes both readings:

• "More budget converts cap-hitters into passes" is not supported here. The conversions happened without using the extra budget.
• "Cap-hitters are degeneration loops" is supported, and /145 is the clean case: given 2.7x the budget it consumed all of it, still returned zero code, at a unique-line ratio of 0.086, by far the lowest in our entire population.

This is also the honest limitation of my own retry design, and I should have seen it before running it. Re-issuing at temp 0.6 with a fresh nonce draws a new sample; it does not resume the old one. So "converted" conflates "more budget helped" with "did not loop this time." The only reason the result is still interpretable is the token counts, which happen to discriminate: finishing under the old ceiling rules out budget as the cause. A cleaner design would pin the seed per arm, which is what @Blackwellboy's run does and mine does not.

The unique-line ratio separates cleanly, which is good news for the loop-detector idea:

population	median uniq-line ratio
retry, did not loop (3)	0.498 to 0.606
retry, looped (/145)	0.086
original cap-hitters (4)	0.320
original non-cap-hitters (160)	0.551

Non-loopers land at 0.5 to 0.6 in both the original and retry populations; the persistent looper is 6x lower. That is a usable online signal, and it is the same discrimination @apollo-mg's compression-ratio stratification and @Blackwellboy's 2.9 to 143x tail ratios found independently. Three different metrics, three stacks, same separation.

What loop recovery is actually worth, scored

Re-scoring the ON arm with the 4 retried samples substituted in, via the official evalplus scorer:

ON arm	HumanEval base	HumanEval+
as-run	94.5	88.4
with cap-hitters resampled	96.3	89.6
OFF arm (unchanged)	95.1	90.9

So a perfect loop-recovery rule is worth +1.2 points on HumanEval+ here, and ON still ends up behind OFF (89.6 vs 90.9).

That is the same conclusion @Blackwellboy reached on NVFP4, in his words "it brings ON to parity on HumanEval+, not ahead," now reproduced at a second quant with a different mechanism (resampling rather than detection). Loop recovery is worth building, and it does not rescue the regime claim.

Practical consequence for the stopping rule

If loops are stochastic rather than problem-inherent, then retry-on-loop-signature is a better rule than raise-the-budget, and it is much cheaper: our three recoveries cost 2k to 7k tokens each, against the 32,768 the budget-raising approach spends before giving up on /145. A detector firing on a low unique-line ratio and re-drawing at the same budget would have recovered 3 of 4 at a fraction of the cost.

Worth noting the cross-stack texture on /47 one more time, since it is now a three-way: benign at Apollo's 2-bit (capped twice, passed both), degenerate on our first Q4 draw (zero code, ratio 0.185), benign on our second Q4 draw (3,767 tokens, ratio 0.600). Same problem, three characters. That is what stochastic looping looks like, and it explains the apparent disagreement without either party being wrong.

Raw: hep_capretry.jsonl alongside the 328-row main JSONL.