scripts: add thinking probes for measuring what controls reasoning

scripts/thinking-probes/README.md

@@ -0,0 +1,105 @@
+# Thinking probes
+
+Three tools for measuring **whether a model actually reasons**, and what
+controls it, on an OpenAI-compatible endpoint.
+
+Separate from `spine-probes/`, which measures integrity behaviour. Different
+question, same house rules: standard library only, no install step, every raw
+response saved.
+
+These exist because of a wrong result. We reported that the `enable_thinking`
+chat-template kwarg was inert on Laguna S 2.1, reasoning from the template
+source, which sets `| default(false)`. The template was read correctly. The
+conclusion was still wrong, because the server supplies that kwarg itself when
+the client omits it, so the default never runs. See
+[#5](https://github.com/TheTom/offlabel/issues/5) for the full correction.
+
+Two habits came out of that and are baked into these tools.
+
+**Reading is not measuring.** `render_check.py` inspects rendered output rather
+than template source, and needs no statistics to do it.
+
+**A finding needs a cell where it cannot happen.** Both sampling tools include a
+control arm where the effect is structurally impossible. If the effect shows up
+there anyway, the instrument is broken and every other number is suspect.
+
+## `render_check.py`: what the template actually renders
+
+The cheapest and most decisive check here. `/apply-template` returns the
+formatted prompt without generating, so it is deterministic and repeats exactly.
+
+```bash
+python3 render_check.py --base http://localhost:8080
+```
+
+It varies one kwarg across four arms (absent, empty object, `false`, `true`) and
+reports which arms render identically. **Test the absent arm.** It is the one
+most people never check and the one most people actually run.
+
+Point it at the template file as well and it renders the same file in stock
+jinja2 and compares:
+
+```bash
+python3 render_check.py --base http://localhost:8080 \
+  --template models/templates/poolside-Laguna-S-2.1.jinja
+```
+
+An arm reported as `SERVER OVERRIDES TEMPLATE` means the server is supplying
+that kwarg itself. In that case the template's default for it is unreachable,
+and reasoning about behaviour from the template alone will mislead you. That is
+exactly the trap we fell into.
+
+Use `--kwarg` for anything other than `enable_thinking`.
+
+## `thinking_ab.py`: does the rendering change behaviour
+
+Interleaved sampling across the same arms, to confirm the rendering has the
+consequence it looks like it should.
+
+```bash
+python3 thinking_ab.py --base http://localhost:8080 --model my-model --seeds 5
+```
+
+Classifies each response three ways, which is the distinction a single field
+cannot make:
+
+| signal | verdict |
+|---|---|
+| `reasoning_content` non-empty | `REASONED_PARSED` |
+| `<think>` markers in content | `REASONED_UNPARSED` |
+| neither | `NO_REASONING` |
+
+`REASONED_UNPARSED` is the important one. Any occurrence means a stack reading
+only `reasoning_content` would report that arm as not firing when it did, so a
+published firing rate from such a stack is measuring the parser as much as the
+model.
+
+## `persona_ab.py`: does a system prompt change it
+
+Crosses a persona system prompt against the kwarg, four cells:
+
+```bash
+python3 persona_ab.py --base http://localhost:8080 --model my-model --seeds 6
+```
+
+The two `false` cells are the control. Thinking is structurally unavailable
+there, so the persona must show no effect. If it appears to, stop and fix the
+measurement before believing the other two cells.
+
+## Design notes, and one warning
+
+Both sampling tools interleave arms round robin, vary the prompt across three
+shapes, send `cache_prompt: false` with a unique nonce per request, and hold
+`max_tokens` fixed across every arm with no retry.
+
+That is not fussiness. Our first pass ran arms in blocks, on one repeated
+prompt, with the cache on, and produced 5/5 in one arm against 0/5 in another.
+It was mostly an artifact. **Manufactured consistency looks exactly like a
+strong result**, and no amount of staring at the numbers reveals it.
+
+Vary the token budget across arms and you have the same problem in another form:
+a difference in outcomes that cannot be attributed to the variable you meant to
+test.
+
+Task shape moves these numbers a lot, so the per-shape breakdown in the output
+is worth reading before drawing a conclusion from the pooled figure.

scripts/thinking-probes/persona_ab.py

@@ -0,0 +1,147 @@
+#!/usr/bin/env python3
+"""Re-measure the persona effect against a baseline we have actually measured.
+
+The original persona result (6/6 fired without a persona, 0/5 with) was taken
+in the enable_thinking-absent condition, on one prompt, in blocks. We since
+established that the absent condition reasons only about half the time and
+that task shape moves it a lot, so the old baseline was not the near-certainty
+it looked like. The effect may well be real; the effect SIZE was measured
+against a constant that is not constant.
+
+This crosses the two factors instead of holding one at an assumed value:
+
+    persona   in {none, senior-staff-engineer}
+    kwarg     in {absent, false}
+
+The false rows are the control. Thinking is structurally unavailable there, so
+persona must show no effect; if it appears to, the measurement is broken rather
+than the persona being powerful.
+
+Interleaved across all four cells, three prompt shapes, cache disabled, unique
+nonce per request, for the same reasons as thinking_ab.py.
+"""
+
+import argparse
+import json
+import re
+import time
+import urllib.request
+from pathlib import Path
+
+PERSONA = "You are a senior staff software engineer."
+
+PROMPTS = {
+    "code-refactor": (
+        "Refactor this Go function so the error is wrapped with context, and "
+        "explain what you changed.\n\n"
+        "func load(p string) ([]byte, error) {\n"
+        "\tb, err := os.ReadFile(p)\n"
+        "\tif err != nil {\n\t\treturn nil, err\n\t}\n"
+        "\treturn b, nil\n}"
+    ),
+    "design-question": (
+        "We store per-tenant config in one Postgres table with a tenant_id "
+        "column. Traffic is growing. What are the trade-offs of splitting to "
+        "schema-per-tenant?"
+    ),
+    "short-factual": (
+        "In one sentence: what does the -race flag do in the Go toolchain?"
+    ),
+}
+
+CELLS = [
+    ("no-persona", "absent"),
+    ("persona", "absent"),
+    ("no-persona", "false"),
+    ("persona", "false"),
+]
+
+THINK_MARKER = re.compile(r"</?think>", re.I)
+
+
+def post(url, body, timeout):
+    req = urllib.request.Request(
+        url, data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.load(r)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--base", required=True)
+    ap.add_argument("--model", required=True)
+    ap.add_argument("--seeds", type=int, default=6)
+    ap.add_argument("--temperature", type=float, default=0.6)
+    ap.add_argument("--max-tokens", type=int, default=1400)
+    ap.add_argument("--timeout", type=int, default=600)
+    ap.add_argument("--out", default="persona-verify.json")
+    args = ap.parse_args()
+
+    plan = [(p, s, persona, kwarg)
+            for s in range(args.seeds)
+            for p in PROMPTS
+            for (persona, kwarg) in CELLS]
+
+    print(f"=== {len(plan)} requests, 4 cells interleaved ===", flush=True)
+    rows = []
+    for i, (pname, seed, persona, kwarg) in enumerate(plan, 1):
+        msgs = []
+        if persona == "persona":
+            msgs.append({"role": "system", "content": PERSONA})
+        msgs.append({"role": "user",
+                     "content": f"{PROMPTS[pname]}\n\n(ref {seed}-{i})"})
+        body = {
+            "model": args.model, "messages": msgs,
+            "max_tokens": args.max_tokens, "temperature": args.temperature,
+            "cache_prompt": False,
+        }
+        if kwarg == "false":
+            body["chat_template_kwargs"] = {"enable_thinking": False}
+
+        t0 = time.time()
+        try:
+            d = post(args.base + "/v1/chat/completions", body, args.timeout)
+            ch = d["choices"][0]
+            content = ch["message"].get("content") or ""
+            reasoning = ch["message"].get("reasoning_content") or ""
+            finish = ch.get("finish_reason")
+            reasoned = bool(reasoning.strip()) or bool(THINK_MARKER.search(content))
+        except Exception as exc:                                # noqa: BLE001
+            content, reasoning, finish, reasoned = f"ERROR: {exc}", "", "", None
+        dt = round(time.time() - t0, 1)
+        print(f"  [{i:>2}/{len(plan)}] {pname:<15} {persona:<10} tk={kwarg:<6} "
+              f"reasoned={str(reasoned):<5} {dt:>6}s reasoning={len(reasoning):>5}",
+              flush=True)
+        rows.append({"prompt": pname, "seed": seed, "persona": persona,
+                     "kwarg": kwarg, "reasoned": reasoned, "seconds": dt,
+                     "reasoning_chars": len(reasoning),
+                     "content_chars": len(content), "finish_reason": finish,
+                     "content": content, "reasoning": reasoning})
+        Path(args.out).write_text(json.dumps(rows, indent=2) + "\n")
+
+    print("\n=== reasoned rate per cell ===")
+    print(f"  {'persona':<11} {'kwarg':<7} {'reasoned':>10}  mean_reasoning_chars")
+    for persona, kwarg in CELLS:
+        rs = [r for r in rows if r["persona"] == persona and r["kwarg"] == kwarg
+              and r["reasoned"] is not None]
+        n = sum(1 for r in rs if r["reasoned"])
+        mr = sum(r["reasoning_chars"] for r in rs) / max(len(rs), 1)
+        print(f"  {persona:<11} {kwarg:<7} {n:>4}/{len(rs):<5} {mr:>18.0f}")
+
+    print("\n=== persona effect WITHIN the absent arm, per prompt shape ===")
+    for pname in PROMPTS:
+        line = f"  {pname:<15}"
+        for persona in ("no-persona", "persona"):
+            rs = [r for r in rows if r["persona"] == persona
+                  and r["kwarg"] == "absent" and r["prompt"] == pname
+                  and r["reasoned"] is not None]
+            n = sum(1 for r in rs if r["reasoned"])
+            line += f"  {persona}={n}/{len(rs)}"
+        print(line)
+
+    print(f"\nwrote {args.out}")
+
+
+if __name__ == "__main__":
+    main()

scripts/thinking-probes/render_check.py

@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+"""Compare what a chat template renders for each value of a template kwarg.
+
+This is the cheapest and most decisive check in this directory, because it
+involves no sampling: llama.cpp's /apply-template returns the formatted prompt
+without generating, so the answer is deterministic and repeats exactly.
+
+It exists because reading a template is not the same as measuring it. A template
+can carry `{%- set enable_thinking = enable_thinking | default(false) -%}` and
+still be handed `true` by the server on every request where the client omits the
+kwarg, which makes the default unreachable. That is only visible in the rendered
+output, and it is what a whole wrong conclusion rested on for us (offlabel#5).
+
+With --template, it also renders the same file locally in stock jinja2 and diffs
+the two. Server and local agreeing means the template decides. Disagreeing means
+the server is supplying the value itself, which is the interesting case.
+"""
+
+import argparse
+import itertools
+import json
+import re
+import urllib.request
+from pathlib import Path
+
+MESSAGES = [{"role": "user", "content": "Say hi."}]
+
+# {% generation %} / {% endgeneration %} are a HuggingFace assistant-masking
+# extension that stock jinja2 cannot compile. They do not affect the branches
+# this tool inspects, so they are stripped for the local render only.
+GENERATION_TAG = re.compile(r"\{%-?\s*(end)?generation\s*-?%\}")
+
+
+def server_render(base, kwarg, value, timeout):
+    body = {"messages": MESSAGES}
+    if value is not None:
+        body["chat_template_kwargs"] = {kwarg: value} if value != {} else {}
+    req = urllib.request.Request(
+        base.rstrip("/") + "/apply-template",
+        data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.load(r).get("prompt", "")
+
+
+def local_render(path, kwarg, value):
+    try:
+        from jinja2 import Environment
+    except ImportError:
+        return None
+    src = GENERATION_TAG.sub("", Path(path).read_text(encoding="utf-8"))
+    tmpl = Environment().from_string(src)
+    kw = {} if value is None else {kwarg: value}
+    return tmpl.render(messages=MESSAGES, add_generation_prompt=True, **kw)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--base", required=True,
+                    help="llama.cpp server root, e.g. http://localhost:8080")
+    ap.add_argument("--kwarg", default="enable_thinking",
+                    help="chat template kwarg to vary (default enable_thinking)")
+    ap.add_argument("--template", default="",
+                    help="optional path to the .jinja file, to also render it "
+                         "locally in stock jinja2 and compare")
+    ap.add_argument("--tail", type=int, default=30,
+                    help="characters of the rendered tail to display")
+    ap.add_argument("--timeout", type=int, default=60)
+    args = ap.parse_args()
+
+    # "absent" sends no chat_template_kwargs at all, which is the arm that shows
+    # what the server does when the client says nothing. It is the one most
+    # people never test, and the one most people actually run in production.
+    arms = [("absent", None), ("empty-kwargs", {}), ("false", False), ("true", True)]
+
+    print(f"=== server render ({args.kwarg}) ===")
+    server = {}
+    for label, value in arms:
+        try:
+            server[label] = server_render(args.base, args.kwarg, value, args.timeout)
+        except Exception as exc:                                # noqa: BLE001
+            print(f"  {label:<13} ERROR {type(exc).__name__}: {exc}")
+            continue
+        tail = server[label][-args.tail:].replace("\n", "\\n")
+        print(f"  {label:<13} len={len(server[label]):>5}  tail={tail!r}")
+
+    print("\n=== which arms render identically? ===")
+    for a, b in itertools.combinations(sorted(server), 2):
+        same = server[a] == server[b]
+        mark = "identical" if same else "DIFFERENT"
+        print(f"  {a:<13} vs {b:<13} {mark}")
+
+    if args.template:
+        print("\n=== local jinja2 render of the same template ===")
+        local = {}
+        for label, value in arms:
+            if label == "empty-kwargs":
+                continue          # identical to absent locally by construction
+            out = local_render(args.template, args.kwarg, value)
+            if out is None:
+                print("  jinja2 not installed, skipping local render")
+                break
+            local[label] = out
+            tail = out[-args.tail:].replace("\n", "\\n")
+            print(f"  {label:<13} tail={tail!r}")
+
+        if local:
+            print("\n=== server versus local, per arm ===")
+            for label in local:
+                if label not in server:
+                    continue
+                agree = server[label][-args.tail:] == local[label][-args.tail:]
+                note = "template decides" if agree else "SERVER OVERRIDES TEMPLATE"
+                print(f"  {label:<13} {note}")
+            print("\n  Any arm marked SERVER OVERRIDES TEMPLATE means the server")
+            print("  supplies that kwarg itself, so the template's default for it")
+            print("  never runs. Do not reason about behaviour from the template")
+            print("  alone in that case.")
+
+
+if __name__ == "__main__":
+    main()