Document audio models using Mimi neural codec

[Copilot]

Comprehensive investigation of Mimi (Kyutai's neural audio codec) usage across the transformers audio model ecosystem.

Findings

Direct Mimi Integration:

• Moshi: Uses Mimi as default audio encoder (model_type="mimi")
• CSM: Uses MimiModel as codec in conversion pipeline

Architecture Inheritance:

• KyutaiSpeechToText: Inherits MoshiPreTrainedModel, MoshiModel; uses MimiConv1dPaddingCache
• Qwen3OmniMoe: Inherits MimiLayerScale for code2wav component
• VibeVoiceAcousticTokenizer: Inherits MimiConv1dPaddingCache for causal convolution

Reusable Components

MimiConv1dPaddingCache → KyutaiSpeechToText, VibeVoiceAcousticTokenizer
MimiLayerScale → Qwen3OmniMoe  
MimiSdpaAttention → Copied to Moshi (with replacements)
MimiModel → Moshi (audio encoder), CSM (codec)

Impact: 5 of 25+ audio models use Mimi/Moshi components. KyutaiSpeechToText shows deepest integration with full architecture inheritance.

Report: MIMI_MOSHI_AUDIO_MODELS_INVESTIGATION.md

Original prompt

I want you to check the audio models in transformers and figureout which models use or inherit mimi from moshi.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.