parakeet-cpp: add plain RNN-T (Transducer) decoder head

parakeet-cpp/CMakeLists.txt

@@ -488,6 +488,9 @@ if (PARAKEET_BUILD_TESTS)
     set(_qvp_ctc_ref       "${PARAKEET_TEST_REF_DIR}/ctc-ref")
     set(_qvp_tdt_ref       "${PARAKEET_TEST_REF_DIR}/tdt-ref")
     set(_qvp_sf_ref        "${PARAKEET_TEST_REF_DIR}/sortformer-ref")
+    set(_qvp_rnnt_q8_gguf  "${PARAKEET_TEST_MODEL_DIR}/stt_ka_fastconformer_hybrid_large_pc.q8_0.gguf")
+    set(_qvp_rnnt_wav      "${PARAKEET_TEST_AUDIO_DIR}/rnnt-ka-16k.wav")
+    set(_qvp_rnnt_ref      "${PARAKEET_TEST_REF_DIR}/rnnt-ref")
 
     add_executable(test-mel
         test/test_mel.cpp
@@ -747,6 +750,30 @@ if (PARAKEET_BUILD_TESTS)
         ARGS     "${_qvp_tdt_q8_gguf}" "${_qvp_fr_multipiece_wav}"
         REQUIRES "${_qvp_tdt_q8_gguf}" "${_qvp_fr_multipiece_wav}")
 
+    # Plain RNN-T greedy decoder parity: <rnnt.gguf> <wav> [ref-dir]. The
+    # rnnt GGUF has no public checkpoint in the fixture set (converted from
+    # a hybrid NeMo checkpoint, e.g. the Georgian model), so the test is
+    # registered with REQUIRES: it shows in `ctest -N` and stays DISABLED
+    # until the fixture files are dropped in place.
+    add_executable(test-rnnt-decoder-parity
+        test/test_rnnt_decoder_parity.cpp
+        src/parakeet_ctc.cpp
+        src/parakeet_engine.cpp
+        src/parakeet_log.cpp
+        src/parakeet_tdt.cpp
+        src/parakeet_eou.cpp
+        src/parakeet_sortformer.cpp
+        src/mel_preprocess.cpp
+        src/sentencepiece_bpe.cpp
+        src/energy_vad.cpp)
+    target_link_libraries(test-rnnt-decoder-parity PRIVATE ggml parakeet-backend-defs)
+    target_include_directories(test-rnnt-decoder-parity PRIVATE include src ggml/include)
+    parakeet_apply_backend_defs(test-rnnt-decoder-parity)
+    parakeet_register_test(test-rnnt-decoder-parity
+        LABEL    "fixture"
+        ARGS     "${_qvp_rnnt_q8_gguf}" "${_qvp_rnnt_wav}" "${_qvp_rnnt_ref}"
+        REQUIRES "${_qvp_rnnt_q8_gguf}" "${_qvp_rnnt_wav}" "${_qvp_rnnt_ref}/token_ids.npy")
+
     add_executable(test-sortformer-parity
         test/test_sortformer_parity.cpp
         src/parakeet_ctc.cpp

parakeet-cpp/PROGRESS.md

@@ -3470,3 +3470,91 @@ doesn't flake the test.
   binary. Surfacing it through downstream addon wrappers
   (e.g. `transcription-parakeet`'s `runStreaming()` JS API) requires
   separate plumbing work on those wrappers — not in this phase.
+
+## Phase 18 — plain RNN-T (Transducer) head  _(done)_
+
+### 18.0 — scope and design decision
+
+Duration-less Transducer checkpoints had no working path: the converter
+mis-tagged them `tdt` and crashed on the absent
+`model_defaults.tdt_durations`; the TDT loader assumes duration logits
+(`V+1+num_durations` joint rows); the EOU decoder is itself a plain
+RNN-T but hard-wires `<EOU>`/`<EOB>` semantics (and its `eou_id`
+fallback `vocab_size-2` would alias real BPE ids on a model without
+those tokens). Target checkpoint family: the RNN-T branch of hybrid
+`EncDecHybridRNNTCTCBPEModel` models, concretely
+`nvidia/stt_ka_fastconformer_hybrid_large_pc` (Georgian).
+
+Design: plain RNN-T is structurally EOU minus the special tokens, so
+the head reuses the EOU predictor/joint runtime (`EouRuntimeWeights`,
+`eou_prepare_runtime`, `eou_decode_window`, `eou_greedy_decode`) with a
+new `EouDecodeOptions.disable_special_tokens` flag instead of adding a
+fourth decoder implementation. In plain mode the greedy inner loop
+breaks only on the transducer blank — matching NeMo greedy RNN-T —
+and `eou_id`/`eob_id` are forced to `-1` (never valid ids).
+
+### 18.1 — converter
+
+`--head {auto,ctc,tdt,rnnt,eou,sortformer}` override;
+`detect_model_type()` now routes RNN-T checkpoints with neither TDT
+durations nor an `<EOU>` label to `rnnt` instead of crashing as `tdt`.
+Emits `parakeet.rnnt.{vocab_size,blank_id,pred_hidden,pred_rnn_layers,
+joint_hidden,max_symbols_per_step}` (blank_as_pad: `blank_id ==
+vocab_size`) + `rnnt.predict.*` / `rnnt.joint.*` tensors; the hybrid's
+CTC aux head (`ctc_decoder.*`) is not exported. Guards: joint output
+rows must equal `vocab+1` (a TDT-shaped joint forced through `--head
+rnnt` fails at convert time, not at decode time), and
+`joint.num_classes` must equal `decoder.vocab_size`.
+
+### 18.2 — loader + Engine dispatch
+
+`ParakeetModelType::RNNT`; `rnnt.*` tensors load into the shared EOU
+weight slots. Decode arms in `Engine::transcribe_samples`, both
+streaming paths (`transcribe_samples_stream`, `StreamSession`
+`process_window` — mirrors the EOU arms with the plain flag and
+persistent decode state), `stream_start` state init, and the CLI.
+`is_transcription_model()` includes RNNT; `model_type()` returns
+`"rnnt"`. Plain RNNT has no native end-pointing signal, so streaming
+sessions take the same opt-in RMS energy VAD as CTC/TDT.
+
+### 18.3 — parity validation
+
+`scripts/dump-rnnt-reference.py` (forces
+`change_decoding_strategy(decoder_type="rnnt")` on the hybrid, dumps
+greedy `token_ids.npy` + transcript + optional `encoder_out.npy`) and
+`test/test_rnnt_decoder_parity.cpp` (CPU vs GPU-offloaded-encoder run,
+plus bit-exact token-id comparison against the NeMo dump; int32 C-order
+`.npy` enforced; wav sample rate validated against the model).
+
+Measured 2026-06-09 on `nvidia/stt_ka_fastconformer_hybrid_large_pc`:
+C++ greedy decode reproduces NeMo greedy token ids **75/75 bit-for-bit**
+on a FLEURS `ka` clip; CPU == Metal at f16; Metal == NeMo at q8_0;
+FLEURS `ka` WER within **0.31 %** of NeMo. The ctest entry
+(`test-rnnt-decoder-parity`, label `fixture`) registers via `REQUIRES`
+and stays DISABLED until the external GGUF + wav + ref dump are placed
+under the fixture dirs (no public checkpoint for this head).
+
+### 18.4 — files touched
+
+- `scripts/convert-nemo-to-gguf.py` — `--head`, rnnt detection +
+  metadata + tensors + shape guards.
+- `scripts/dump-rnnt-reference.py` (new) — NeMo greedy reference dump.
+- `src/parakeet_ctc.{h,cpp}` — `ParakeetModelType::RNNT`, rnnt GGUF
+  loader arm, model summary.
+- `src/parakeet_eou.{h,cpp}` — `disable_special_tokens` plain mode;
+  RNNT accepted by `eou_prepare_runtime` with `-1` sentinel ids.
+- `src/parakeet_engine.cpp` — offline + Mode 2 + Mode 3 dispatch,
+  `stream_start` init, energy-VAD comment scope.
+- `src/main.cpp` — CLI decode arm + usage text.
+- `test/test_rnnt_decoder_parity.cpp` (new) + `CMakeLists.txt`
+  fixture vars and `parakeet_register_test` entry.
+
+### 18.5 — follow-ups
+
+- Streaming-window decode (Mode 2/3) has no dedicated parity harness
+  yet; it reuses the EOU state machinery verified in §12 but a
+  windowed-vs-offline token-id cross-check on a long clip would close
+  the gap.
+- Decode is scalar-CPU (like EOU). If the fused TDT Metal decoder
+  (§15) ever generalises to `num_durations == 0`, plain RNNT can ride
+  the same graph path.

parakeet-cpp/README.md

@@ -1,6 +1,6 @@
 # parakeet.cpp
 
-**Parakeet** (NVIDIA FastConformer ASR family, CC-BY-4.0) ported to [`ggml`](https://github.com/ggml-org/ggml). Pure C++ inference on **CPU** and **GPU** (Metal / Vulkan / OpenCL); no Python, PyTorch, or onnxruntime at runtime. One **`parakeet::Engine`** loads **CTC**, **TDT**, **EOU**, or **Sortformer** GGUFs and dispatches by metadata.
+**Parakeet** (NVIDIA FastConformer ASR family, CC-BY-4.0) ported to [`ggml`](https://github.com/ggml-org/ggml). Pure C++ inference on **CPU** and **GPU** (Metal / Vulkan / OpenCL); no Python, PyTorch, or onnxruntime at runtime. One **`parakeet::Engine`** loads **CTC**, **TDT**, **RNNT**, **EOU**, or **Sortformer** GGUFs and dispatches by metadata.
 
 ## Supported checkpoints
 
@@ -17,22 +17,24 @@
 
 Encoder topology is selected from GGUF metadata (`conv_norm_type`, causal subsampling, chunked-limited attention, etc.), so EOU shares the same C++ graph path as CTC/TDT where weights allow.
 
+Plain **RNN-T** (`parakeet.model.type = "rnnt"`) covers duration-less Transducer heads — e.g. the RNN-T branch of a hybrid `EncDecHybridRNNTCTCBPEModel` checkpoint (`--head rnnt` in the converter; the hybrid's CTC aux head is ignored). There is no public checkpoint in the fixture set; the head is verified against NeMo greedy decoding on `nvidia/stt_ka_fastconformer_hybrid_large_pc` (Georgian) via `test-rnnt-decoder-parity` + `scripts/dump-rnnt-reference.py`. The decoder reuses the EOU predictor/joint runtime (scalar CPU) with special-token handling disabled.
+
 ## API overview
 
 | Surface | Role |
 |---------|------|
-| `Engine::transcribe` | One-shot wav → text (CTC / TDT / EOU) or segments (Sortformer) |
+| `Engine::transcribe` | One-shot wav → text (CTC / TDT / RNNT / EOU) or segments (Sortformer) |
 | `Engine::transcribe_stream` | Mode 2: full encode once, stream segments |
 | `Engine::stream_start` → `StreamSession` | Mode 3: live duplex / cache-aware chunks |
 | `Engine::diarize` / `diarize_start` | Sortformer offline / live streaming (v1: sliding-history; v2.1: speaker-cache / AOSC) |
 | `transcribe_with_speakers` | Sortformer + ASR → attributed transcript |
 
-EOU streaming segments expose `is_eou_boundary`. **`StreamEvent`** (optional callbacks) covers end-of-turn (EOU) and VAD-style signals (Sortformer threshold, optional energy VAD on CTC/TDT). **`Engine::backend_device`** / **`backend_name`** reflect the backend actually used after the load-time cascade.
+EOU streaming segments expose `is_eou_boundary`. **`StreamEvent`** (optional callbacks) covers end-of-turn (EOU) and VAD-style signals (Sortformer threshold, optional energy VAD on CTC/TDT/RNNT). **`Engine::backend_device`** / **`backend_name`** reflect the backend actually used after the load-time cascade.
 
 ## Pipeline
 
 ```
-wav → log-mel → FastConformer encoder → CTC / TDT / EOU / Sortformer decoder
+wav → log-mel → FastConformer encoder → CTC / TDT / RNNT / EOU / Sortformer decoder
 ```
 
 Each GGUF bundles weights, mel filterbank, and tokenizer as needed.
@@ -107,6 +109,8 @@ python scripts/convert-nemo-to-gguf.py \
 
 **Important:** for non-default checkpoints set **`--hf-repo`** (e.g. `nvidia/parakeet-tdt-0.6b-v3`) — the script otherwise defaults to the CTC repo and may download the wrong weights. Use `scripts/download-all-models.sh` to prefetch `.nemo` files.
 
+The head is auto-detected from `cfg['target']`; **`--head {ctc,tdt,rnnt,eou,sortformer}`** overrides it. Hybrid transducer+CTC checkpoints (`EncDecHybridRNNTCTCBPEModel`) export their plain RNN-T branch as `--head rnnt` (also the auto-detect result when the checkpoint has neither TDT durations nor an `<EOU>` token); the hybrid's CTC aux head is ignored.
+
 Default **`--quant`** is **`q8_0`**. Use **`f16`** for parity-calibrated harnesses (noise from q8 swamps NeMo FP32 references).
 
 ### Quantization tiers (CTC 0.6B, M4 Air CPU)
@@ -150,7 +154,7 @@ CMake builds the main binary as target **`parakeet-cli`** with **`OUTPUT_NAME pa
 
 **Synopsis:** `parakeet --model <.gguf> (--wav <.wav> | --pcm-in <.raw>) [options]`
 
-The GGUF picks the engine (CTC / TDT / EOU transcription vs Sortformer diarization). Optional **`--diarization-model <sortformer.gguf>`** adds speaker labels when **`--model`** is a CTC/TDT GGUF (“who said what”).
+The GGUF picks the engine (CTC / TDT / RNNT / EOU transcription vs Sortformer diarization). Optional **`--diarization-model <sortformer.gguf>`** adds speaker labels when **`--model`** is a CTC/TDT/RNNT GGUF (“who said what”).
 
 | Topic | Flags |
 |------|--------|
@@ -259,6 +263,8 @@ python scripts/dump-ctc-reference.py --wav test/samples/jfk.wav
 python scripts/dump-tdt-reference.py --wav test/samples/jfk.wav
 python scripts/dump-eou-reference.py --wav test/samples/jfk.wav
 python scripts/dump-sortformer-reference.py --wav test/samples/diarization-sample-16k.wav
+# rnnt: no fixture wav in-repo; point --wav at your own 16 kHz clip
+python scripts/dump-rnnt-reference.py --nemo-model <hybrid.nemo> --wav <clip-16k.wav>
 
 cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
 cmake --build build -j

parakeet-cpp/include/parakeet/attributed.h

@@ -1,6 +1,6 @@
 #pragma once
 
-// Speaker-attributed transcription: Sortformer segments + ASR text per slice (CTC/TDT/EOU).
+// Speaker-attributed transcription: Sortformer segments + ASR text per slice (CTC/TDT/RNNT/EOU).
 
 #include "export.h"
 #include "engine.h"

parakeet-cpp/include/parakeet/engine.h

@@ -3,7 +3,7 @@
 // Loaded GGUF inference: transcribe, stream, diarize, and backend metadata behind one Engine class.
 //
 // Loads weights once; subsequent calls pay mel + encoder + decode only. Model kind (CTC, TDT,
-// EOU, Sortformer) comes from GGUF metadata.
+// RNNT, EOU, Sortformer) comes from GGUF metadata.
 //
 // Transcription:
 //   - transcribe / transcribe_samples — one-shot wav or PCM to text.
@@ -230,7 +230,7 @@ class PARAKEET_API Engine {
 
     const EngineOptions & options() const;
 
-    // "ctc", "tdt", "eou", or "sortformer", reflecting the
+    // "ctc", "tdt", "rnnt", "eou", or "sortformer", reflecting the
     // parakeet.model.type metadata of the loaded GGUF.
     std::string model_type() const;
 

parakeet-cpp/include/parakeet/parakeet.h

@@ -6,8 +6,8 @@
 //   <parakeet/cli.h>         - parakeet_cli_main C entry point
 //   <parakeet/log.h>         - parakeet_log_set host log sink
 //   <parakeet/engine.h>      - Engine + EngineOptions / EngineResult
-//                                   (CTC, TDT, EOU, Sortformer behind one
-//                                   class)
+//                                   (CTC, TDT, RNNT, EOU, Sortformer behind
+//                                   one class)
 //   <parakeet/streaming.h>   - StreamingOptions / StreamingSegment /
 //                                   StreamSession + cross-engine
 //                                   StreamEvent + VadState +

parakeet-cpp/include/parakeet/streaming.h

@@ -17,7 +17,7 @@ namespace parakeet {
 // Optional StreamEvent callback: VadStateChanged and EndOfTurn alongside segment text.
 //
 // EOU models emit EndOfTurn when `<EOU>` fires. Sortformer emits VadStateChanged from
-// speaker_probs vs threshold. CTC/TDT can use optional RMS EnergyVad when enabled.
+// speaker_probs vs threshold. CTC/TDT/RNNT can use optional RMS EnergyVad when enabled.
 
 enum class VadState : int {
     Unknown  = 0,
@@ -67,12 +67,12 @@ struct StreamingOptions {
     // Optional; nullptr disables StreamEvent delivery (segment-only streaming).
     StreamEventCallback on_event = nullptr;
 
-    // Energy-VAD fallback. When true, CTC / TDT sessions will compute a
-    // simple RMS-thresholded VAD over the input PCM and fire
+    // Energy-VAD fallback. When true, CTC / TDT / RNNT sessions will compute
+    // a simple RMS-thresholded VAD over the input PCM and fire
     // `StreamEventType::VadStateChanged` events on transitions. Always-on
     // for sessions whose underlying engine (EOU, Sortformer) has its own
     // native VAD source -- those engines' events take priority. Default
-    // off; opt-in for CTC/TDT consumers that want VadState events.
+    // off; opt-in for CTC/TDT/RNNT consumers that want VadState events.
     bool  enable_energy_vad = false;
 
     // Energy-VAD knobs (dB-scale; applies only when enable_energy_vad).
@@ -109,7 +109,7 @@ struct StreamingSegment {
     // segment whose token list is empty (defensive default).
     bool   starts_word = true;
 
-    // EOU-only: true when this segment ends on `<EOU>`. For CTC/TDT use StreamEvent
+    // EOU-only: true when this segment ends on `<EOU>`. For CTC/TDT/RNNT use StreamEvent
     // EndOfTurn via `on_event` instead; those engines leave this flag false here.
     bool   is_eou_boundary = false;
     float  eot_confidence  = 0.0f;