🎙️ Custom Voice TTS with MOSS-TTS

Clone your own voice and generate high-quality speech using MOSS-TTS — an open-source, production-grade Text-to-Speech family from MOSI.AI and the OpenMOSS team. Upload a short audio recording of your voice, type any text, and the model synthesizes it in your voice — no training required.

---

📖 Overview

This Colab notebook provides a complete, ready-to-run pipeline for zero-shot voice cloning. It:

1. Converts your voice recording from .m4a (or any format) to .wav using pydub + ffmpeg
2. Loads the MOSS-TTS-Local-Transformer model and processor directly from HuggingFace
3. Builds a voice-clone conversation using processor.build_user_message() with your reference audio path
4. Runs model.generate() with configurable VQ codebook depth and token budget
5. Decodes the output and saves it to moss_tts_out/ as a .wav file

The 1.7B Local Transformer model was chosen because it fits within Google Colab free-tier GPU memory (T4 ~15 GB VRAM) while still producing high-fidelity, natural-sounding output.

---

✨ Features

• 🎤 Zero-shot voice cloning — no fine-tuning, just a short reference clip
• 🌍 Multilingual — English, Chinese, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
• ⚡ Auto attention backend — selects flash_attention_2, sdpa, or eager based on your hardware automatically
• 🔊 Native sampling rate output via processor.model_config.sampling_rate
• 📁 Accepts any audio format — .m4a, .mp3, .wav all supported (auto-converted)
• 💾 Saves output to moss_tts_out/abhiraam_cloned_0.wav

---

🗂️ Project Structure

Custom-Voice-MOSS-TTS/
│
├── Copy of tts.ipynb            # Main Colab notebook
├── README.md                    # This file
├── abhiraams.m4a                # Your input reference voice recording
├── abhiraams.wav                # Auto-converted reference (generated by Cell 1)
└── moss_tts_out/
    └── abhiraam_cloned_0.wav    # Generated output audio

---

🚀 Quick Start

1. Open in Colab

Click the badge above or visit:

https://colab.research.google.com/drive/1-PyfFw_msz4p7nybv4YjFuSbeU3UAfIz?usp=sharing

2. Switch to GPU Runtime

Runtime → Change runtime type → T4 GPU → Save

3. Upload Your Voice

Drag your .m4a or .wav file into the Colab file panel (left sidebar), or use:

from google.colab import files
files.upload()

4. Run All Cells

Runtime → Run all
Output will be saved to moss_tts_out/abhiraam_cloned_0.wav

---

📓 Notebook Walkthrough

Cell 1 — Install Dependencies & Convert Audio

!pip install pydub ffmpeg-python -q
!apt-get install -y ffmpeg -q

from pydub import AudioSegment
audio = AudioSegment.from_file("abhiraams.m4a")
audio.export("abhiraams.wav", format="wav")
print("Converted to abhiraams.wav")

Installs pydub and ffmpeg, then converts the reference audio to .wav — required by the MOSS-TTS processor.

---

Cell 2 — Load Model, Clone Voice, Save Output

Attention backend auto-selection:

def resolve_attn_implementation() -> str:
    if device == "cuda" and flash_attn installed and dtype in {fp16, bf16}:
        if GPU compute capability >= 8:       # A100, H100
            return "flash_attention_2"
    if device == "cuda":
        return "sdpa"                          # T4, V100
    return "eager"                             # CPU

Hardware	Backend Used
A100 / H100 with flash_attn	flash_attention_2
T4 / V100 (standard Colab GPU)	sdpa
CPU	eager

Model loading:

MODEL_ID = "OpenMOSS-Team/MOSS-TTS-Local-Transformer"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
processor.audio_tokenizer = processor.audio_tokenizer.to(device)

model = AutoModel.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    attn_implementation=attn_implementation,
    torch_dtype=dtype,
).to(device).eval()

Building the voice-clone conversation:

The reference audio is passed as a local file path string inside a list to the reference= argument:

conversations = [
    [
        processor.build_user_message(
            text=synth_text,
            reference=[ref_audio_path],   # must be a path/URL string, not a tensor
        )
    ]
]

Tokenizing inputs:

batch = processor(conversations, mode="generation")
input_ids      = batch["input_ids"].to(device)
attention_mask = batch["attention_mask"].to(device)

Generating speech:

outputs = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    max_new_tokens=4096,
    n_vq_for_inference=32,
)

Decoding and saving:

messages = processor.decode(outputs)

for i, message in enumerate(messages):
    audio = message.audio_codes_list[0]
    torchaudio.save(
        save_dir / f"abhiraam_cloned_{i}.wav",
        audio.unsqueeze(0),
        processor.model_config.sampling_rate
    )

---

⚙️ Key Parameters

Parameter	Default	Description
max_new_tokens	4096	Output length limit. ~12.5 tokens = 1 second of audio
n_vq_for_inference	32	VQ codebook layers. Lower = faster + lower bitrate. Try 8 or 4 for speed
torch_dtype	bfloat16 (GPU) / float32 (CPU)	Model precision
attn_implementation	auto-detected	Attention backend (flash_attention_2 / sdpa / eager)

Duration reference:

max_new_tokens = 125   →  ~10 seconds
max_new_tokens = 375   →  ~30 seconds
max_new_tokens = 800   →  ~64 seconds
max_new_tokens = 4096  →  ~5 minutes (full budget, GPU required)

---

🎙️ Tips for Best Voice Clone Quality

Tip	Detail
Length	10–30 seconds of reference audio is ideal
Clarity	Clean speech only — no background noise, music, or echo
Format	Any format works; auto-converted to WAV
Language	Matching reference and synthesis language gives best results
reference= argument	Must be a file path string in a list, not a waveform tensor

---

🧠 Model Details

Property	Value
Model ID	OpenMOSS-Team/MOSS-TTS-Local-Transformer
Parameters	~1.7 Billion
Architecture	MossTTSLocal (Local Attention Transformer)
Audio Tokenizer	Loaded automatically via AutoProcessor
Token Rate	12.5 tokens / second of audio
trust_remote_code	True (required)

---

📊 MOSS-TTS Family Comparison

Model	Params	Min VRAM	Quality	Colab Free Tier
MOSS-TTS-Nano	100M	CPU only	Good	✅
MOSS-TTS-Local-Transformer ← this notebook	1.7B	~8 GB	Great	✅ T4
MOSS-TTS Delay (flagship)	8B	~24 GB	Best	❌ crashes

---

🛠️ Troubleshooting

Session crashed / Out of Memory

• Verify GPU runtime is enabled: Runtime → Change runtime type → T4 GPU
• Reduce max_new_tokens to 800
• Reduce n_vq_for_inference to 8 or 4
• Do not use OpenMOSS-Team/MOSS-TTS (8B) on Colab free tier — it will OOM

Slow or rate-limited HuggingFace downloads

Add your token to Colab Secrets (🔑 left sidebar):

• Name: HF_TOKEN
• Value: your token from huggingface.co/settings/tokens

ffmpeg not found error

!apt-get install -y ffmpeg

Poor cloning quality / wrong voice

• Use a longer, cleaner reference recording (15–30 sec)
• Ensure reference= receives a list containing a file path string: [ref_audio_path]
• Avoid background noise, music, or reverb in your reference audio

transformers class not found

!pip install --upgrade transformers -q

Then restart the runtime and re-run all cells.

---

📦 Dependencies

Package	Purpose
pydub	Audio format conversion
ffmpeg-python + ffmpeg	Audio decoding backend
torch + torchaudio	Model inference and audio I/O
transformers	AutoModel, AutoProcessor for MOSS-TTS

All installed automatically inside the notebook.

---

🌐 Supported Languages

Language	Language	Language
🇺🇸 English	🇨🇳 Chinese	🇯🇵 Japanese
🇰🇷 Korean	🇩🇪 German	🇫🇷 French
🇷🇺 Russian	🇧🇷 Portuguese	🇪🇸 Spanish
🇮🇹 Italian

Language is detected automatically from synth_text — no manual flag needed.

---

⚠️ Ethical Use Notice

• ✅ Clone your own voice for portfolios, demos, or accessibility tools
• ✅ Experiment with TTS research and voice synthesis
• ❌ Do not clone another person's voice without their explicit consent
• ❌ Do not use for impersonation, fraud, or any deceptive purpose

---

🙏 Credits

• OpenMOSS Team — MOSS-TTS model family
• MOSI.AI — Research and infrastructure
• HuggingFace — Model hosting

---

👤 Author

Abhiraam S
Final-year Computer Science Student, Bangalore Institute of Technology

Built to explore AI voice synthesis for real-world portfolio and demo use cases.