Benchmarks tell you if a model can solve the problem. They don't tell you what it's like to drive.
offlabel is a set of per-model operating guides: practical, evidence-backed notes on how a model actually
behaves once you're past the leaderboard: does it hold a line under pressure or fold? Does thinking mode help
or quietly sabotage the answer? Will it over-refuse benign work, or under-refuse risky work? Does it stay honest
across a long multi-turn task, or capitulate to a false premise and then paper over it?
None of that shows up in a pass@1 number. It shows up when you actually use the thing. This repo is the "actually use the thing" notes, written down so you don't have to rediscover them yourself.
Every model gets a shareable card: an at-a-glance infographic with the operating dials (thinking, tools, sampling) and a green/red Trust Map of where it holds vs folds under pressure. Full write-up in each guide.
Gemma 4 family (per-size, because each size behaves differently, see the family overview):
Static benchmarks assume a model is a fixed function: same input, same output, one true score. Modern models break that assumption. They're non-deterministic, many public benchmarks are contamination-prone, and the axes that matter for actually deploying a model (integrity under pressure, sycophancy, calibration, multi-turn coherence, tool reliability) are mostly untested by standard leaderboards. A model can look great on a benchmark and still be the wrong tool for your task, or the right tool used the wrong way (wrong sampling config, thinking mode left on when it should be off, wrong harness for its tool-calling format).
Each guide here is built from held-out, hands-on behavioral testing: scenarios the model hasn't seen before, run head-to-head against a trusted baseline where possible, plus direct observation of how the model behaves in practice. It's opinionated and specific: we tell you what to do differently, not just what the model scored.
Every file in models/ follows the same shape (see TEMPLATE.md for the full spec):
- Cheat sheet: five lines, top of the file: what to reach for it, what to avoid, thinking policy, tool/agent fit, and the sharpest trust boundary.
- Numbered sections: envelope (best-at / not-for), thinking policy with dose-response where we have it, prompting/persona notes, tools & agents, sampling & serving, and the trust map (what it holds the line on vs. what it folds on under pressure).
- Confidence & scope on every claim: how much testing backs it and what config (quant/build/date) it was
tested on, so guidance doesn't outlive its evidence. Untested axes are marked
⬚ backlog, not silently assumed.
Every guide also has a matching card in cards/<name>.html: a single shareable infographic (see
cards/_reference.html for the exact layout to clone; brand.md for the palette). The card is the
80/20 summary: cheat sheet + trust map in one image. The guide is the full case.
Every guide and every card is scored against the same ten axes, so models are comparable across the whole set instead of each write-up inventing its own frame:
| # | Axis | What it answers |
|---|---|---|
| 1 | Vibe & voice | personality, tone, writing style, weird habits |
| 2 | Refusal calibration | over-refusal (blocks benign work) vs under-refusal (allows risky work) |
| 3 | Sycophancy & spine | pushes back vs capitulates/flatters; false-premise resistance; integrity under pressure |
| 4 | Hallucination & calibration | invents facts/bugs; expresses uncertainty vs overconfident; declines unknowables |
| 5 | Instruction-following & coherence | sticks to system prompt/format; multi-turn drift |
| 6 | Thinking / reasoning | control, dose-response (helps/hurts per axis), token cost |
| 7 | Tools & agents | native vs generic harness fit, tool-arg reliability, loop/recovery |
| 8 | Bias & fairness | political/cultural/etc. systematic leanings |
| 9 | Jailbreak / safety robustness | filter-bypass resistance |
| 10 | Serving & config | sampling, quant, serving gotchas |
Coverage per axis is tagged ✅ measured (formally scored head-to-head), 🟡 observational (noted from use, not formally scored), or ⬚ backlog (not tested yet, we say so instead of guessing).
Recurring patterns that show up across more than one model (thinking-mode trade-offs, RL-training artifacts,
quantization fidelity, etc.) live in patterns.md so they don't get re-derived per guide.
Every guide names its scope: what quant/build/date it was tested on, and what was not tested. Guidance here
is a snapshot, not a permanent verdict. Models get patched, fine-tuned, and re-quantized, and a guide gets a
changelog entry when it's re-verified. If a guide's behavioral data is thin, its card says so (DRAFT in the
footer), better an honest draft than false confidence.
If you find offlabel useful, you can support it via GitHub Sponsors or BTC:
BTC: bc1qsfaaf6mkz2yxx2vavg2n0zgsf3qj25uh94t83rwuq7de67dey05sc3tgjx
Commercial support: For behavioral model evaluation, red-teaming, or fine-tune assessment engagements, DM @no_stp_on_snek on X.
Apache License 2.0, see LICENSE.
Copyright 2026 Tom Turney.