QVAC-19261: tts-cpp: add Parler-TTS engine (mini-v1 / large-v1, CPU)#92
Open
pratiknarola-t wants to merge 3 commits into
Open
QVAC-19261: tts-cpp: add Parler-TTS engine (mini-v1 / large-v1, CPU)#92pratiknarola-t wants to merge 3 commits into
pratiknarola-t wants to merge 3 commits into
Conversation
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
New self-contained engine tts_cpp::parler: Flan-T5 description encoder (RMSNorm, relative-position bias, no attention scaling, gated-GELU) with per-description cross-KV precompute, MusicGen-style delay-pattern decoder LM (9 summed codebook embeddings, 9 LM heads, HF-faithful logits processor / min-new-tokens / stopping, exact-erf GELU, sinusoidal positions, token-major KV slab), and a DAC 44.1 kHz codec decoder (RVQ from_codes, snake activation with the +1e-9 guard, unpadded conv-transpose + view trim, F32 im2col convs). Single GGUF per model (parler.* metadata drives mini/large; no code branching) produced by scripts/convert-parler-to-gguf.py with two-sided tensor-name completeness checks and weight-norm folding; the T5 unigram tokenizer (precompiled charsmap + Metaspace + Viterbi) ships in the GGUF. Verified vs the HF PyTorch reference (scripts/dump-parler-reference.py fixtures): tokenizer ids exact (12-case corpus incl. unicode), T5 max_abs 1.3e-4, decoder prefill + teacher-forced steps max_abs <= 5.8e-4 with per-codebook argmax equality, delay/logits-processor decisions exact vs a trace of the real HF classes, DAC 121-124 dB SNR, and the full greedy e2e token trace matches HF exactly (429/429 steps; wav 121.5 dB SNR). Direct-vs-sched dispatch is bit-identical, ASAN is clean, and the full existing ctest suite stays green (83/83 runnable). The only shared-code touch is tts-cli model-family detection (parler.arch sniffed before the chatterbox tokenizer fallback) plus the --description flag; a standalone parler-cli mirrors supertonic-cli. F16 conversion keeps the T5 encoder in F32 (Flan-T5 activations overflow the f16 range).
- Repair weight-norm parametrizations from the safetensors ground truth after loading: transformers' sharded low-mem load leaves parametrizations.weight.original0/1 at init values for large-v1's DAC (mini-v1 single-file loads are unaffected; both v1 checkpoints ship byte-identical DAC weights), so without this the large fixtures encode a corrupted codec. - Prepend a cloning no-op logits processor so output_logits stays raw: the ParlerTTS EOS gate mutates scores in place, and for large-v1 (no min_new_tokens processor cloning before it) the -inf EOS mask was stamped into the dumped "raw" logits.
pratiknarola-t
force-pushed
the
qvac-19261-parler-tts
branch
from
dfaebde to
2618a23
Compare
- reset cross-KV state before the fallible re-encode work in parler_encode_description and invalidate the engine's description cache when encoding fails, so a failed re-encode can never leave a stale-but-plausible cross-attention state behind - validate max_frames in the engine: values <= n_codebooks cannot yield a single audio frame and now throw up front (with a regression test) - honor --output-sample-rate on the parler tts-cli path via the existing sinc resampler - wrong-arch engine test: skip honestly when the fixture is not staged and assert the arch-check message when it is - drop the unused n_gpu_layers engine option; record delayed_len from the reconstructed token sequence in the fixture dump metadata
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new self-contained
tts_cpp::parlerengine: description-conditioned TTS forparler-tts/parler-tts-mini-v1andparler-tts/parler-tts-large-v1on CPU. mini/large differences are pure GGUF metadata (parler.*keys) — no code branching.Pipeline: Flan-T5 encoder (description → cross-attention K/V, precomputed once and cached per description; RMSNorm, relative-position bias, no attention scaling, gated-GELU) → delay-pattern decoder LM (9 summed codebook embeddings, 9 LM heads, MusicGen-style stagger with HF-faithful EOS gating / min-new-tokens / stopping, exact-erf GELU, sinusoidal positions, token-major KV slab) → DAC 44.1 kHz codec (RVQ
from_codes, snake activation incl. the+1e-9guard, unpaddedconv_transpose_1d+ view trim, F32-im2col convs) → mono PCM.Additive everywhere: the only shared-code touch is
tts-clifamily detection (parler.archsniffed before the chatterboxtokenizer.ggml.tokensfallback) plus a--descriptionflag; a standaloneparler-climirrorssupertonic-cli. Graph dispatch uses the sharedsched_dispatchdual path (fresh graph per pass, per the single-use contract).Conversion:
scripts/convert-parler-to-gguf.pyemits ONE GGUF per model (T5 + decoder + DAC + T5 unigram tokenizer with precompiled charsmap), with two-sided tensor-name completeness checks and weight-norm folding.--dtype f16keeps the entire T5 encoder F32 (Flan-T5 activations overflow the f16 range; ggml's f16 mul_mat converts activation rows to f16 for the dot product → NaN), plus norms/biases/alphas/positional table/DAC.Verification (fixtures from
scripts/dump-parler-reference.py, HF PyTorch reference)TTS_CPP_FORCE_SCHED=1Sampled decoding (model default: temp 1.0, top-k 50) terminates via natural EOS — the delay-pattern stopping that is known-broken in the TTS.cpp reference implementation (their issue #50).
Upstream quirks found (and worked around in the dump script — not engine code)
parametrizations.weight.original0/1at init values for large-v1's DAC (single-file mini is unaffected; both v1 checkpoints ship byte-identical DAC weights). Stock HF parler-large actually runs with that corrupted codec on modern stacks; the GGUF ships the true weights, and the reference dump repairs the parametrizations before dumping.ParlerTTSLogitsProcessormutates scores in place, contaminatingoutput_logitswhen nothing clones before it (large has no min-new-tokens processor) — the dump prepends a cloning no-op.Out of scope / follow-ups
scripts/setup-ggml.shpins a bundled-ggml SHA that predates the lavasr custom ops, so the bundled dev build of master is currently broken independent of this PR (worked around locally by checking out speech HEAD).