fix: skip firstTokenLogProbThreshold when promptTokens are set

[alan890104]

Problem

Using promptTokens in DecodingOptions causes intermittent empty transcription results, particularly with distilled/turbo model variants (e.g. large-v3-turbo). This is the root cause of #372.

Root Cause Analysis

When promptTokens are provided, the decoder input sequence becomes:

[<|startofprev|>] + [prompt_tokens...] + [<|sot|>] + [lang] + [task] + [notimestamps]

The prompt tokens shift the decoder's KV cache state, causing the first content token's logprob to occasionally drop below firstTokenLogProbThreshold (default: -1.5). When this happens:

1. isFirstTokenLogProbTooLow is set to true
2. The decoding loop immediately breaks — producing zero content tokens
3. avgLogProb computes to 0.000 (no real tokens to average over)
4. DecodingFallback triggers with reason "firstTokenLogProbThreshold"
5. Temperature is increased and decoding retries, but the first token may still fail the threshold
6. After exhausting all fallback attempts, the final result is empty

Why this only affects WhisperKit

firstTokenLogProbThreshold is a WhisperKit-specific quality gate — it does not exist in:

• OpenAI's original Whisper (whisper/decoding.py) — only uses logprob_threshold (avg over full segment), no_speech_threshold, and compression_ratio_threshold
• whisper.cpp — faithfully ports the original three thresholds without adding a first-token check

The original Whisper design lets the decoder run to completion and evaluates quality over the entire segment via avgLogprob. This is robust to prompt-induced shifts in the first token's distribution because subsequent tokens compensate. WhisperKit's early abort on the first token prevents this self-correction.

Why it's intermittent

The first token logprob depends on the interaction between prompt token content and audio content. For certain audio segments, the prompt conditioning pushes the first token just below -1.5; for others it stays above. This creates a non-deterministic failure pattern.

Why turbo models are more affected

Distilled/turbo variants (e.g. large-v3-v20240930) have fewer decoder layers, making them more sensitive to changes in the conditioning context. The reduced decoder capacity has less room to absorb the distributional shift from prompt tokens.

Fix

Skip the firstTokenLogProbThreshold check when promptTokens are present (options.promptTokens == nil guard). This is a one-line change in TextDecoder.swift.

Safety: The existing logProbThreshold (avg over full segment, default -1.0) and compressionRatioThreshold remain active as quality gates, matching the original Whisper behavior. Truly bad segments will still be caught and retried via temperature fallback.

Related

• Fixes Using promptTokens causes the Transcription to return empty result. #372

[Copilot]

Pull request overview

This PR addresses intermittent empty transcription results when DecodingOptions.promptTokens is used by disabling WhisperKit’s firstTokenLogProbThreshold early-abort check in that mode (root cause of #372).

Changes:

• Skip the firstTokenLogProbThreshold check when promptTokens are provided.
• Add a regression unit test ensuring transcription is non-empty with promptTokens even under a strict firstTokenLogProbThreshold.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
Sources/WhisperKit/Core/TextDecoder.swift	Disables the first-token logprob early-abort when options.promptTokens is non-nil to prevent empty outputs.
Tests/WhisperKitTests/UnitTests.swift	Adds regression coverage for prompting with a strict first-token threshold (Issue #372).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

This change makes firstTokenLogProbThreshold effectively a no-op whenever options.promptTokens is non-nil. That’s a behavior change for an existing public option, so it should be documented (e.g., in DecodingOptions docs / README) to avoid confusing callers who set a strict threshold expecting it to be enforced.