Bug: CosyVoice3 produces garbled speech on torch >= 2.7 (required by RTX 50-series)

[TOMUIV]

Bug: CosyVoice3 produces garbled speech on torch >= 2.7 (required by RTX 50-series)

Environment

• GPU: NVIDIA RTX 5060 (Blackwell sm_120)
• OS: Windows 11
• torch: 2.11.0+cu128
• transformers: 5.6.2
• CosyVoice commit: ace7c47 (latest main)
• Model: Fun-CosyVoice3-0.5B-2512 (official weights from modelscope)

Steps to reproduce

from cosyvoice.cli.cosyvoice import AutoModel
model = AutoModel(model_dir="pretrained_models/Fun-CosyVoice3-0.5B-2512", fp16=True)

for i, j in enumerate(model.inference_zero_shot(
    "八百标兵奔北坡，北坡炮兵并排跑。",
    "You are a helpful assistant.<|endofprompt|>希望你以后能够做的比我还好呦。",
    "./asset/zero_shot_prompt.wav",
    stream=False
)):
    torchaudio.save(f"out_{i}.wav", j["tts_speech"], model.sample_rate)

Output is unintelligible noise. When transcribed back with FunASR-Nano, the result is gibberish (Japanese kana, repeated syllables).

Root cause

CosyVoice3 LM inference (cosyvoice/llm/llm.py:459) uses forward_one_step() which calls Qwen2ForCausalLM.forward() through the transformers library. In transformers >= 5.x, Qwen2Attention.forward() uses ALL_ATTENTION_FUNCTIONS.get_interface(), which replaced the direct scaled_dot_product_attention call from 4.x. RMSNorm, KV cache, and rotary embedding APIs also changed.

These API-level differences cause minute numerical drift. CosyVoice3 generates speech tokens autoregressively — after 100+ steps, the accumulated drift collapses output into noise.

Why CosyVoice2 works

I verified that CosyVoice2 runs successfully on the same environment. The reason is architectural: CosyVoice2 uses cosyvoice/transformer/encoder.py:forward_chunk() — a hand-written chunk-based transformer that calls torch.nn.functional.scaled_dot_product_attention directly, bypassing transformers Qwen2 wrappers entirely.

Why this matters for RTX 50-series users

RTX 50-series GPUs require torch >= 2.7 (sm_120 support). The official CosyVoice3 requirements.txt pins torch 2.3.1 (no sm_120). Users are forced onto torch >= 2.7 + transformers >= 5.x, creating a hard incompatibility that does not exist for 40-series and older GPUs.

Related issues

• Cosyvoice3 example.py例子生成的句子完全乱读 #1692 — same symptoms (closed, no response)
• 创空间的CosyVoice3很好用，但是开源的webui效果很差 #1741 — same symptoms (closed, no response)
• 5060ti 16g 运行cosyVoice2 无法启动. 求教!! #1318 — CosyVoice2 on 50-series (torch too old, fixed by upgrading to 2.8+)

Suggested fix

1. Backport CosyVoice3 inference to use forward_chunk() like CosyVoice2, or
2. Make CosyVoice3LM.inference() compatible with transformers 5.x Qwen2 API, or
3. Document minimum transformers version constraint.

[TOMUIV]

Update after further testing:

1. CosyVoice2 does NOT work either. I tested it on the same environment (torch 2.11 + transformers 5.6 + RTX 5060) and it produces 28 seconds of noise — STT transcribed it back as a single word "Okay." The issue body's "Why CosyVoice2 works" section is incorrect.

2. Why v2 and v3 share the same root cause. The class hierarchy is:

TransformerLM (line 34)        → inference() uses forward_chunk() — v1 only
Qwen2LM(TransformerLM) (line 257) → inference() OVERRIDES parent → uses forward_one_step() → Qwen2ForCausalLM
CosyVoice3LM(Qwen2LM) (line 664) → inherits Qwen2LM.inference() unchanged

Both v2 and v3 instantiate Qwen2LM, so both go through forward_one_step() → Qwen2ForCausalLM.forward() → transformers 5.x native Qwen2 implementation. Neither uses forward_chunk(). Confirmed via:

Qwen2LM.inference is TransformerLM.inference  →  False
'inference' in Qwen2LM.__dict__               →  True
CosyVoice3LM.inference is Qwen2LM.inference   →  True

3. What actually works: CosyVoice-300M (v1). It uses TransformerLM.inference() → forward_chunk() → custom encoder in cosyvoice/transformer/encoder.py, bypassing transformers library entirely. Tested on the same environment — zero-shot output transcribed back as correct Chinese.

4. Suggested fix: v2/v3's Qwen2LM.inference() needs to either (a) use forward_chunk() path like v1, or (b) be made compatible with transformers 5.x Qwen2 API.