ThinkDiff

Watch reasoning models think through the same problem — side by side.

Every lab publishes its own reasoning-model evals. Nobody shows the traces together in a way you can actually read. When o-style, R1, QwQ, and Gemini-Thinking all attack the same problem — do they reason the same way, or does one backtrack three times while another walks straight to the answer?

ThinkDiff renders reasoning traces as animated parallel streams. Press ▶ Play and watch each model think in real time, with cognitive events auto-highlighted:

• 🟠 backtracks ("wait, actually…") — where a model catches its own mistake
• 🔵 verification ("let me check: 2+3=5") — where it double-checks
• 🔴 dead-ends — abandoned approaches

Plus per-model stats (steps, words, backtracks, "reconsider %") and a reasoning-overlap table showing which models think most alike.

Quick start (zero config)

python -m venv .venv && .venv\Scripts\python -m thinkdiff build
# open reports/thinkdiff.html and press Play

Ships with a real trace corpus: 4 model personas on 2 classic "trap" problems (bat-and-ball, count-the-7s). The terse model falls for both traps and gets them wrong; the reasoner backtracks and recovers — you see the difference:

bat-and-ball: A bat and a ball cost $1.10 ...
  ✓ r1-reasoner            13 steps, 1 backtracks, 2 checks -> $0.05
  ✓ gemini-flash-thinking   5 steps, 0 backtracks, 0 checks -> $0.05
  ✓ qwq-verbose            16 steps, 0 backtracks, 1 checks -> $0.05
  ✗ gpt-mini-terse          3 steps, 0 backtracks, 0 checks -> $0.10   # the trap

Add your own traces

Drop a JSON file in traces/:

{
  "problem_id": "my-puzzle",
  "question": "…",
  "gold": "42",
  "traces": {
    "deepseek-r1": {"final": "42", "trace": "First… Wait, let me reconsider… Let me verify… So 42."},
    "qwq-32b":     {"final": "42", "trace": "…"}
  }
}

Paste the raw <think> content of any reasoning model into trace. ThinkDiff segments it into steps, detects backtracks/verification, scores correctness against gold, and adds it to the side-by-side view. This works with R1, QwQ, Kimi, Gemini-Thinking — any model that exposes its chain of thought.

What it measures

• step segmentation — splits a trace into thought units on sentence/line boundaries
• cognitive events — cue-based detection of backtrack / verify / dead-end
• reconsider ratio — fraction of steps spent second-guessing (a proxy for how much a model "struggles" on a problem)
• reasoning overlap — token-Jaccard between each model pair's full trace

Why it's interesting

"Is o-style reasoning actually different from R1, or just longer?" is one of the liveliest open questions in 2026. ThinkDiff turns it into something you can watch and measure, and the trace corpus itself is a reusable artifact for reasoning analysis.

Status

v0.1 — trace segmenter, cognitive-event detection, correctness scoring, cross-model similarity, animated self-contained HTML (no server needed to view), tests + CI. Roadmap: paste-from-clipboard trace import, live provider fetch, reasoning-as-DAG view.

MIT licensed.