Score what decides advancement, not just delivery

[bradburch]

The problem, on the bench

The headline score averaged four delivery dimensions and nothing else. Same transcript (fluent, structured, brute-force-only, three escalating hints, couldn't state a complexity), same model:

Rubric	Result
Old (delivery-only, flat mean)	4.25 → green in the UI
New	lean_no — with all four delivery dimensions still at 4

The user's own data says the same thing: OneSchema scored their highest marks (3.0, 3.0, 2.0) and is dead; Cloneable scored their lowest (1.75) and is active.

Three structural causes

1. Nothing ever asked about advancement. The schema requested scores and prose. advancement is now a discrete 4-point forced choice (strong_no/lean_no/lean_yes/strong_yes), elicited independently and explicitly not derived from the scores.
2. The schema forbade scoring substance. additionalProperties: false over four fixed keys meant the detailed signal already written into product_sense.md / tech_deep_dive.md could never reach a score. Dimensions are now parsed from each overlay's ## Scored dimensions section — a new round type is a new .md file, no Swift change.
3. Averaging compressed dimensions kills discrimination. Anchored 1-5 bands + an explicit anti-leniency instruction. overallScore stays as a secondary trend line, comparable within a round type only.

What the research did and didn't support

20 sources, 25 claims adversarially verified, 11 survived. Recorded in the DefaultPrompts header.

• Not supported: whether advancement is conjunctive or compensatory. Claims in both directions were refuted, so no weighting formula was added. (The Kuncel meta-analysis often cited here actually argues for a flat average over a holistic gestalt.)
• Authored judgment, and labelled as such: no primary rubric survived for recruiter_screen, system_design, product_sense, or tech_deep_dive. Coding rests on one mock-interview platform's scorecard.
• Supported: discrete verdict recorded separately from ratings; anchored scales (.35 vs .26); job-specific content over generic dimensions.

Also included

• process_notes — timestamped extraction of what the interviewer said about process/next steps/timeline, in every round type. Surfaced per company in the Pipeline (newest round first) and above the debrief in a session. Guarded against inventing next steps from a polite sign-off.
• TranscriptArtifacts — strips Whisper non-speech markers at the Store write boundary (the funnel both live-stop and crash recovery pass through). 11% of segments in the real database were [BLANK_AUDIO], silence variants, chunk-cut brackets, or >> speaker markers. Migration v5 applies identical rules to existing rows.
• Live progress for re-run debriefs — state lives on AppEnvironment because MainWindow's tab switch destroys SettingsView; as view @State progress vanished on tab switch while the run continued, and returning re-enabled the button (second concurrent run).
• Pipeline cells navigate to their session.

Bugs found by verifying against reality

Two classes of bug that swift test cannot see, both caught by real calls:

• The Messages API rejects minimum/maximum on integer types (400). Mocked tests passed against a schema that would have failed every debrief.
• A real debrief in the database was stored as all-zeros — the rubric says 1-5 and nothing enforced it. The band is now enforced in CoachingResult's decoder (rejects rather than clamps, so the session stays retryable).

Testing

• 104 unit tests, 0 failures (was 79)
• CoachingIntegrationTests — real API calls; the only check that the built schema is one the API accepts, and that the rubric elicits what it asks for
• Migrations v3/v4/v5 verified against a copy of a real 11-session database (rows preserved; 4015 → 3560 segments, 0 artifacts)
• Pipeline navigation verified by driving the running app via accessibility

Reviewer notes

• The four scored dimensions were hardcoded in three places that enforce the same contract by different mechanisms: ClaudeClient.outputSchema (JSON schema), OpenAICompatibleClient.formatAppendix (prose — local servers can't be trusted with json_schema), and the prompt markdown. They drift independently; scores being [String: Int] means a local model returning wrong keys still decodes, so there's an explicit key-set check.
• PromptStore.ensureDefaults() only writes files that don't exist — editing DefaultPrompts alone will not update an existing install.

Known, not smoothed over

• process_notes returned [] once, early, on a transcript full of stated process. Correct 5/5 times since; not reproducible, and not explained. Possible intermittent.
• The verdict rationale came back as the literal word "placeholder" once, never again.
• The re-run progress UI is unit-tested at the service layer but has not been watched through a full real run (11 × ~35s, and it overwrites existing debriefs).

🤖 Generated with Claude Code

[bradburch]

Review round: 10 verified findings, 9 fixed, 1 declined with evidence

High-effort workflow review (28 agents, independent verifier per finding). The first attempt reported "no findings survived verification" — that was an infrastructure failure, not a clean bill of health: all four finder agents died on a session limit and produced zero candidates. Re-ran after the reset.

The one that mattered — would have broken every existing install

dimensions(for:) returned [] on any install that had ever launched Debrief. ensureDefaults() never overwrites, and the old seeded base.md has no ## Scored dimensions heading → empty scores schema → every debrief stored with a 0.0 average, resurrecting the exact all-zeros bug this PR's decoder was added to prevent. The local-LLM path failed every debrief outright. "Re-run debriefs on current rubric" would have spread it across all history.

Why the original verification missed it: every test seeds into a fresh temp dir (works), and I'd hand-regenerated my own live prompts — masking it on the only machine that could have shown it. I documented this exact gotcha in CLAUDE.md and still walked into it.

ensureDefaults now upgrades a stale builtin, preserving the old file as <name>.md.pre-dimensions.bak, and dimensions(for:) throws rather than returning []. Verified end-to-end against the real pre-PR prompts on a real install: all 7 upgraded, all 7 backed up, every round resolves non-empty dimensions.

Also fixed

Finding	Fix
Zero-row insertSegments defeated the orphan guard → empty-transcript session still got coached, LLM confabulating a debrief	returns count; finalize throws into existing cleanup
bareMarkerSentence ate real speech — "No audio.", "Pause. Let me think about that."	narrowed to the two markers observed bare in real data
Stop reported "Stopped after 11 of 11" and hid failures	progress is the authority, not Task.isCancelled
retryAllPending inherited break-on-cancel; a stray URLError.cancelled silently ended the run	keyed on Task.isCancelled, not error type
Pipeline reveal never scrolled (onChange installed after the value was set)	task(id:)
ForEach over process notes keyed on a non-unique timestamp	index
Re-coach failure invisible on a complete session	badge keys on status, not overallScore == nil
processNotes used decodeIfPresent while both clients declare it required	strict decode

Declined, with evidence

The finding that the paren rule eats spoken asides ("we shipped it (about 200ms p99)"). All 10 distinct parenthesized spans in the real 11-interview database are non-speech annotations — (Laughter) (door opens) (eerie music) (upbeat music) (laughing) (sniffing) (scissors snipping) (indistinct) (inaudible) (clicking) — and zero are speech. Whisper inherits subtitle convention where parens mean sound events. The proposed narrowing would leak five of those; the counter-examples were invented, not observed. Rule stays broad, with the evidence recorded in the code.

Known and measured, not fixed

process_notes recall misses on ~2 in 7 runs against a transcript that explicitly states the process. This is model behaviour, not a branch. The instruction was rebalanced — it spent three sentences on not inventing notes and one on capturing them, which suppressed recall — and moved out of the trailing field list. The integration test is documented as a recall probe that can flake, rather than weakened into something that cannot fail. The feature is additive: when it misses you are where you are today; when it fires you get next steps you'd otherwise dig for.

114 unit tests, 0 failures (was 79 at branch start).