eval(speaker): measured accuracy delta for multi-exemplar + negative-exemplar features#1493
Merged
Conversation
…exemplar features Adds a corpus-gated A/B harness that measures the two speaker-identity features merged in 2026-07 (#1488 multi-exemplar voiceprints, #1487 negative-exemplar veto) against the legacy single-average matcher, on real cached AMI/VoxCeleb per-quality embeddings, at the certified operating point (auto-bar 0.92 + margin 0.12). Harness (Tests/TranscriptedCoreTests/SpeakerExemplarDeltaEvalTests.swift) enrolls mature profiles through the real SpeakerDatabase (real EMA + SpeakerExemplarPolicy) and scores held-out degraded cross-condition trials WITH (bestSimilarity + veto) vs WITHOUT (average-only cosine, no veto) using the real production policies. XCTSkips unless SPEAKER_EVAL_QMATRIX_DIR points at the (~11 GB, not in-tree) fingerprint cache. Headline (docs/speaker-eval-exemplar-delta-2026-07.md): - Multi-exemplar: AMI genuine recall +10.9pp (54.9->65.9%), correct silent auto-recognition +8.1pp (1.6->9.7%) — but +12 silent mislabels (false-auto 0.0->1.6%) the legacy matcher had at 0. The certified "0 false-auto" no longer holds on degraded audio once best-of-exemplars scoring is live. - Negative veto: removes 46% of repeat wrong-matches in-room (AMI) but only 12% cross-condition (VoxCeleb); ~2% owner-collateral. - Recommendation: keep both; retune the auto gate for the exemplar path (margin vs the average rep, not the best exemplar) to restore 0-false-auto without losing recall. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
r3dbars
added a commit
that referenced
this pull request
Jul 7, 2026
…ansport The negative-exemplar veto (#1487) was "sound but regime-limited" (eval in #1493): after a correction it removed 46% of repeat wrong-matches in-room (AMI) but only 12% cross-condition (VoxCeleb). Root cause: a single rejected in-room sample and a later telephone/VoIP sample of the same rejected impostor sit too far apart in embedding space for the raw cosine to clear the 0.80 veto floor. The positive side already closed this gap with multi-exemplar voiceprints (#1488); the negative side — one rejected embedding — had no analog. Give the negative side the same multi-condition treatment (eval rec #3): compare the candidate against each rejected sample AND against that sample transported along the profile's own observed condition shifts (unit(exemplar) − average). Channel/condition shifts are largely speaker-independent, so a rejected sample shifted by a condition the profile has actually seen approximates the same rejected voice returning in that other condition. Owner-safe by construction: - transport only along +(exemplar − average) — real, forward, profile-observed conditions; bidirectional/synthetic directions were measured to leak owner-collateral and are not used. - a profile with no positive exemplars derives nothing → reduces exactly to the raw maxNegativeSimilarity → single-condition profiles unchanged. - the 0.80 floor and the ≥positiveSimilarity owner gate are untouched; transport only widens which impostor returns are caught, never lowers the floor. Real qmatrix eval (SpeakerExemplarDeltaEvalTests): AMI cross-condition vetoed-among-re-match 0.350 → 0.374 (+7.0% rel) with owner-collateral flat (2.17% → 2.32%, +9 of 5773 checks). VoxCeleb (single-utterance corpus, no condition structure) byte-identical, incl. owner-collateral — no regression. Pushing VoxCeleb's 12% further was investigated and rejected: every mechanism that moves it trades owner-collateral ~1:1, violating the #1493 guardrail. See docs/speaker-eval-negative-veto-cross-condition-2026-07.md. Refs #1493. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A measurement PR (no production behavior change): quantifies the real accuracy delta of the two speaker-identity features that merged into
mainin 2026-07 — multi-exemplar voiceprints (#1488) and negative-exemplar veto (#1487) — which passed CI but had no measured precision/recall delta proving they moved speaker-naming accuracy.Adds
Tests/TranscriptedCoreTests/SpeakerExemplarDeltaEvalTests.swift, a corpus-gated A/B harness, and the findings docdocs/speaker-eval-exemplar-delta-2026-07.md(+ raw result JSON).How it measures
Enrolls mature, multi-condition profiles through the real
SpeakerDatabase(real EMA blend + realSpeakerExemplarPolicyexemplar maintenance) on real cached AMI + VoxCeleb per-quality fingerprints (256-d WeSpeaker means from the real diarizer, the sameTools/SpeakerEvalHarnessqmatrix used by the ladder). Held-out degraded cross-condition trials (opus_8k / tel_g711 / noisy / reverb / mp3_16) are scored two ways at the certified operating point (auto-bar 0.92 + margin 0.12):SpeakerVectorMath.bestSimilarity(candidate, average, exemplars)+ the realSpeakerNegativeExemplarPolicyveto (both features live).cosineSimilarity(candidate, average)only, no veto (the pre-feat(speaker): negative exemplars — learn "this voice is NOT this person" from corrections #1487/Multi-exemplar voiceprints for speaker matching #1488 single-average matcher; reproduces the ladder's certified 0 false-auto).The auto-name decision is the real
SpeakerNamingPolicy.shouldAutoAcceptin both arms. The harness XCTSkips unlessSPEAKER_EVAL_QMATRIX_DIRpoints at the (~11 GB, not in-tree) fingerprint cache, so CI stays green.Headline delta
Multi-exemplar (#1488) — real recall win, real safety cost. On AMI (147 profiles, 741 degraded trials):
The 12 false-autos are all distinct AMI speakers matched via one lucky exemplar clearing both the 0.92 bar and the 0.12 margin (sim 0.93–0.97, margin 0.12–0.27, mature). The certified "0 false-auto" no longer holds on degraded audio once best-of-exemplars scoring is live. Cross-checked by an independent Python re-implementation (reproduces recall and the 12-count to the digit) and a per-case audit.
Negative veto (#1487) — sound but regime-limited. Removes 46% of repeat wrong-matches in-room (AMI) but only 12% cross-condition (VoxCeleb) — the ≥0.80 veto floor is defeated by the same condition-gap multi-exemplar was built to close. ~2% owner-collateral.
Recommendation
Keep both features (gains are large and real), but retune the auto gate for the exemplar path before treating "0 false-auto" as still true: compute the best-vs-second margin against the profile's average representative, not its best exemplar, so an impostor clearing via one lucky exemplar fails the margin while a genuine owner (close on both) still passes. Separately, give the negative veto a cross-condition representation so it fires in the VoIP/telephone regime. Full analysis, caveats (worst-case slice; fingerprint-level not full-pipeline), and reproduction commands in the doc.
Tests
swift test --filter SpeakerNamingSimulationRunnerTests— 7/7 (existing regression, confirms Core builds with both features onmain).swift test --filter SpeakerExemplarDeltaEvalTests— passes (~3.4s), writes the result JSON.Note: an independent Codex methodology review was attempted but blocked by a Codex usage limit (resets 2026-07-09); methodology was instead self-verified via the independent Python implementation + per-case false-auto audit.
🤖 Generated with Claude Code