parakeet-cpp: add plain RNN-T (Transducer) decoder head#90
Open
rends-east wants to merge 3 commits into
Open
Conversation
Adds a `rnnt` model type alongside CTC/TDT/EOU/Sortformer for plain RNN-T (Transducer) ASR -- e.g. the Transducer branch of a hybrid EncDecHybridRNNTCTCBPEModel (Georgian stt_ka_fastconformer_hybrid_large_pc). Plain RNN-T is structurally EOU minus the <EOU>/<EOB> tokens, so it reuses the EOU predictor/joint runtime (EouRuntimeWeights), eou_prepare_runtime, and eou_decode_window: - converter: detect hybrid / `--head rnnt`; emit parakeet.rnnt.* metadata + rnnt.predict.*/rnnt.joint.* tensors (TDT minus the duration head). - loader: ParakeetModelType::RNNT; load rnnt.* into the shared EouWeights slot; eou_prepare_runtime forces eou_id/eob_id = -1 for rnnt. - decoder: EouDecodeOptions.disable_special_tokens makes the greedy loop break only on the transducer blank (matches NeMo greedy). - dispatch: Engine::transcribe_samples + CLI (main.cpp) decode arms. - test: test-rnnt-decoder-parity + scripts/dump-rnnt-reference.py. Verified on nvidia/stt_ka_fastconformer_hybrid_large_pc: the C++ greedy decoder reproduces NeMo greedy RNN-T token IDs bit-for-bit (75/75 on a FLEURS ka clip; CPU == ggml/Metal graph), and FLEURS ka WER matches NeMo within 0.31%. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the plain RNN-T head through transcribe_samples_stream (windowed) and the duplex StreamSession decode so an rnnt model never falls into the CTC branch (which would dereference the absent CTC head). Mirrors the EOU streaming arms with disable_special_tokens and reuses the persistent eou_state; stream_start initialises eou_state for rnnt too. Not required for the non-streaming target model but removes a crash-on-misuse edge if an rnnt GGUF is fed to the streaming API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
Post-review hardening of the plain RNN-T head; no decode-semantics changes.
- CMake: register test-rnnt-decoder-parity via parakeet_register_test with
REQUIRES fixture paths (shows in `ctest -N`, DISABLED until the external
GGUF + wav + NeMo ref dump are dropped in place) instead of an
unregistered build-only target.
- Parity harness: enforce int32/C-order `.npy` (reject int64/fortran dumps
loudly), hard-fail when an explicitly requested ref-dir cannot be loaded
(was WARN + exit 0), validate the wav sample rate against the model, add
missing <cctype>, and fix the usage text -- the RNN-T decode is scalar
CPU in both runs; n_gpu_layers only moves the encoder.
- Converter (both this repo's script and the vendored monorepo copy stay
hunk-identical): guard --head rnnt on non-transducer checkpoints, assert
joint rows == vocab+1 (a TDT-shaped joint fails at convert time, not at
decode time), assert joint.num_classes == decoder.vocab_size, warn when
NeMo's uncapped max_symbols is baked to 10, fix the wrong
"label vocab + blank" comment, document the rnnt flavour in the module
docstring.
- Docs: README (engine list, --head, dump-script list, rnnt prose),
PROGRESS.md Phase 18 journal entry with the 2026-06-09 parity numbers,
CLI usage ("five engine families"), model_type() contract comment,
umbrella/streaming/attributed header enumerations, energy-VAD comment
scope (plain RNNT takes the CTC/TDT opt-in RMS VAD), capture-parity
harness model-type label.
Verified: clean Metal build; test-rnnt-decoder-parity registers Disabled
without fixtures; model-free ctest passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a plain RNN-T (Transducer) decoder head to
parakeet-cpp/, alongside the existing CTC / TDT / EOU / Sortformer heads — GGUF model typernnt. This enables duration-less Transducer checkpoints, in particular the RNN-T branch of NeMo hybridEncDecHybridRNNTCTCBPEModelmodels (verified onnvidia/stt_ka_fastconformer_hybrid_large_pc, Georgian), which currently have no working path:detect_model_type()classifies a duration-less RNN-T checkpoint astdtand crashes on the missingmodel_defaults.tdt_durations;num_durations=5, mismatching a plain RNN-T joint (V+1outputs);<EOU>/<EOB>handling (itseou_idfallbackvocab_size-2would alias real BPE tokens).How
Plain RNN-T is structurally EOU minus the special tokens, so the head reuses the EOU predictor/joint runtime instead of adding a new decoder:
--head {auto,ctc,tdt,rnnt,eou,sortformer}override + hybrid checkpoint detection; emitsparakeet.rnnt.*metadata andrnnt.predict.*/rnnt.joint.*tensors (the hybrid's CTC aux head is not exported). Guards: joint rows must equalvocab+1andjoint.num_classesmust equaldecoder.vocab_size, so a TDT-shaped joint fails at convert time.ParakeetModelType::RNNT;rnnt.*tensors load into the shared EOU weight slots;eou_id/eob_idforced to-1;EouDecodeOptions.disable_special_tokens— the greedy loop breaks only on the transducer blank, matching NeMo greedy RNN-T semantics. Decode is scalar CPU (like EOU);n_gpu_layersoffloads the encoder;Engine::transcribe_samples, both streaming paths (transcribe_samples_stream,StreamSession::process_window,stream_start), and the CLI;test-rnnt-decoder-parity(CPU vs GPU-offloaded-encoder run, plus bit-exact token-id comparison against a NeMo dump from the newscripts/dump-rnnt-reference.py), following the PROGRESS.md parity methodology. Registered viaparakeet_register_testwithREQUIRESfixture paths: it shows inctest -Nand stays DISABLED until the external GGUF + wav + reference dump are dropped in place (no public checkpoint for this head).Verification
Measured 2026-06-09 on
nvidia/stt_ka_fastconformer_hybrid_large_pc: the C++ greedy decoder reproduces NeMo greedy RNN-T token IDs 75/75 bit-for-bit on a FLEURSkaclip; CPU == Metal at f16; Metal == NeMo at q8_0; FLEURSkaWER matches NeMo within 0.31%.After rebasing onto current
master: clean build with-DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON;test-rnnt-decoder-parityregisters as Disabled without fixtures; model-free ctest passes. The parity numbers above were measured at the original base; I can re-run them against a current build on request.Downstream wiring for
packages/transcription-parakeet(qvac monorepo) is a separate follow-up PR: tetherto/qvac#3228.Developed with AI assistance (disclosed in the commit trailers); reviewed and verified against NeMo references by the author.