Every serious buyer of an AI system asks the same question: how do we know it's right? On production LLM systems for insurance carriers and a UK mortgage broker, I answered it with machinery rather than reassurance — recorded-and-replayed LLM calls, golden-sample regression suites, and semantic judges hardened until they fail for the right reasons — so a 99.55% precision figure is re-checked by machinery on every prompt change, not just quoted from launch day.
| Value | What it is |
|---|---|
| 48× | Faster test suite |
| $0 | Replay cost per run |
| 100+ | Golden production samples |
| 99.55% | Precision, guarded |
Classic software testing assumes the same input gives the same output. LLMs break that assumption: ask twice, get two differently worded answers — both possibly right.
The first-generation tests on the systems I inherited asserted substrings of model output, so they passed or failed on phrasing rather than meaning. Each run also called the live model: slow, expensive, and flaky enough that engineers had learned to re-run failures until they went green.
As lead engineer at a US AI consultancy, I owned this problem on an underwriting platform used by insurance carriers — where prompts kept changing and a precision figure clients relied on had to keep being true. An earlier document-extraction project for a UK mortgage broker posed the same question in miniature.
The discipline is to split every output into what must be exact and what must be equivalent, then test each accordingly. Numbers — discount percentages, surcharge calculations — are checked against golden production samples with explicit numeric tolerance. Reasoning is checked by an LLM judge that asks whether the argument matches the ground truth, not whether the wording does.
| Step | What happens |
|---|---|
| Record once | Real LLM calls captured as VCR cassettes; a new golden sample is ~10 lines of JSON |
| Replay free | Full suite runs in 5 seconds, not 240 — with $0 in API spend |
| Judge meaning | Numeric tolerance on figures, an LLM-as-judge on the reasoning |
| Verify live | Two-region integration harness with flake triage and a metrics ledger |
Underneath sits an evaluation framework with three assertion types — rule-based, semantic, and AI-reasoning — configured per company and per line of business, so each insurer's suite states exactly what "correct" means for its book. I migrated the whole test estate off substring assertions onto this framework, and the golden-sample suite replays 100+ real production cases on every prompt change.
An LLM-as-judge is itself non-deterministic, and a lenient judge quietly rubber-stamps regressions. I hardened the judge questions until they failed for the right reasons — including an explicit "reject filter-out reasoning" rule, so a model that reached the correct answer by dismissing the evidence still fails the test.
Replayed cassettes are fast but frozen: change a prompt and the recording answers a question nobody asks any more. Cassette invalidation is therefore a validation gate — when the request no longer matches, the test forces a fresh recording rather than passing against a world that no longer exists.
Live integration tests against real models will sometimes fail for reasons that aren't bugs. The two-region EC2 harness triages known-noise failures, re-runs suspected flakes once, and appends every run to a metrics ledger — so flake rates are tracked numbers, not folklore, and a genuine regression stands out.
On the mortgage broker's payslip pipeline, I measured model quality as the number of human edits needed to reach ground truth — weighted per wrong field, missing pair, and missing line item. Swapping a model or prompt became a number moving, not an argument about vibes.
| Metric | Result |
|---|---|
| Feedback loop | 240 s → 5 s per test run (48× faster), at $0 in LLM calls on replay |
| Regression coverage | 100+ golden production samples; adding one is ~10 lines of JSON |
| Precision held | 99.55% precision on explicit discounts and surcharges (96.14% overall), guarded by a 100+-sample golden regression suite that runs on every prompt change — a property of the suite, not a launch statistic |
| Live verification | Dual-region integration runs (82 tests in ~8 minutes) with flake triage and a per-run metrics ledger |
Both engagements sit under NDA, so the companies are anonymised. The insurance figures come from the platform's own documentation and test infrastructure; the human-edit-count metric is from the mortgage-broker project's evaluation harness. Happy to walk through the test architecture in as much depth as confidentiality allows.
A designed PDF version of this case study is in this repo: 03-testing-nondeterministic-ai.pdf.
Freelance AI engineer — Expert-Vetted on Upwork (top 1%), 100% Job Success over 70+ projects, $400K+ earned, 5,750+ hours billed. I build production LLM systems for regulated industries: insurance, finance, law.