Skip to content

README.md

Latest commit

 

History

History
199 lines (148 loc) · 7.81 KB

README.md

File metadata and controls

199 lines (148 loc) · 7.81 KB

MusicGen - Conditional Genre Music Generation

Authors: Daria Alexandrova, Kamilya Shakirova

A systematic investigation into how decoding strategies (temperature, top‑k, top‑p) affect the quality, diversity, and prompt adherence of music generated by Meta's MusicGen model. This repository includes a complete pipeline for fine‑tuning on custom audio, running controlled parameter sweeps, evaluating with automated metrics (FAD, CLAP, repetition, diversity), and conducting blind human listening studies.

Overview

This project extends the original AudioCraft MusicGen model by:

  • Fine‑tuning on an out‑of‑distribution genre — the ULTRAKILL industrial breakcore soundtrack — using LoRA adapters to keep training tractable on consumer hardware.
  • Sweeping decoding parameters independently to avoid confounding effects: temperature (0.5–1.5), top‑k (50–500), top‑p (0.85–0.99), and greedy argmax.
  • Measuring four complementary metrics for each condition:
    • FAD (Fréchet Audio Distance) — overall audio quality.
    • CLAP similarity — prompt adherence.
    • Repetition score & loop ratio — intra‑clip looping behaviour.
    • Diversity score — spread of samples from the same prompt.
  • Validating findings with a human study using a custom web‑based pairwise comparison interface.

The main conclusion: temperature = 1.0 (unmodified sampling) provides the best balance of quality and prompt adherence, while greedy decoding collapses into repetitive noise and should be avoided.

Metrics Explained

Metric Description Direction
FAD Fréchet distance between VGGish embeddings of generated and reference audio. ↓ lower = better
CLAP similarity Cosine similarity between CLAP text and audio embeddings. ↑ higher = better prompt match
Repetition score Mean off‑diagonal cosine similarity of mel‑spectrogram frames. ↓ lower = less looping
Diversity score Mean pairwise L2 distance between mean‑mel embeddings of samples. ↑ higher = more varied outputs

Project Structure

MusicGen/
├── config/
│   └── default.yaml               # Central configuration for sweeps & evaluation
├── data/
│   ├── ultrakill/                 # Reference audio files
│   ├── ultrakill_manifests/       # Train/valid/test JSONL manifests
│   │   ├── train/data.jsonl
│   │   ├── valid/data.jsonl
│   │   └── test/data.jsonl
│   └── ultrakill_prompts.jsonl    # Evaluation prompts
├── results/
│   ├── audit/                     # Greedy decoding invariance audit
│   ├── sweep_ultrakill/           # Generated audio & manifest.json
│   ├── metrics_ultrakill/         # Per‑condition metric JSONs & summary.json
│   └── human_study/               # Pairwise comparison tasks & responses
├── src/
│   ├── data/
│   │   ├── local_dataset.py       # Build manifests from local audio folder
│   │   └── pipeline.py            # FMA dataset download & manifest creation
│   ├── human_study/
│   │   ├── generate_pairs.py      # Generate pairwise tasks from manifest
│   │   └── viewer.html            # Web interface for blind listening tests
│   ├── metrics/
│   │   ├── evaluate_all.py        # Run all metrics for a sweep
│   │   ├── prompt_adherence.py    # CLAP‑based prompt similarity
│   │   ├── repetition.py          # Repetition & diversity scores
│   │   └── analysis.py            # Correlation heatmap & analysis
│   ├── audit_decoding.py          # Verify greedy temperature‑invariance
│   ├── evaluate.py                # FAD computation wrapper
│   ├── generate.py                # Single‑shot generation script
│   ├── run_experiments.py         # Controlled sweep over decoding params
│   └── train.py                   # Fine‑tune MusicGen (LoRA / layer‑wise)
├── requirements.txt
└── README.md

Installation

Python 3.11 is required. AudioCraft has strict dependency constraints; follow the two‑step install carefully.

# Install base dependencies
pip install -r requirements.txt

# Install AudioCraft separately (avoids torch/xformers conflicts)
pip install --no-deps "audiocraft @ git+https://github.com/facebookresearch/audiocraft.git"

For LoRA fine‑tuning, also install peft:

pip install peft

! MusicGen‑small requires ~8 GB VRAM for a batch size of 2 with 30‑second clips. Reduce --duration or use --batch_size 1 if memory‑constrained.

Usage

1. Prepare a Custom Dataset

Place your audio files (.flac, .wav, .mp3, etc.) and optional .txt sidecars in a single folder. If a sidecar is missing, a default label is used.

python -m src.data.local_dataset \
    --input_dir ./my_dataset \
    --output_dir ./data/custom_manifests \
    --default_label "ULTRAKILL OST, industrial metal breakcore, Heaven Pierce Her" \
    --split 0.8 0.1 0.1

This creates train/, valid/, test/ folders with data.jsonl manifests.

2. Fine‑tune MusicGen

LoRA is suitable for small datasets (rank=16, alpha=32). Training on an RTX 4060 with 3 epochs takes ~7 hours for 30‑second clips.

python -m src.train \
    --manifest_dir ./data/ultrakill_manifests \
    --audio_dir ./my_dataset \
    --output_dir ./trained_model \
    --lora \
    --epochs 5 \
    --batch_size 2 \
    --duration 30 \
    --lr 1e-4

Without --lora, only the last 4 transformer layers are unfrozen.

3. Run Decoding Sweep

Generate audio under different sampling strategies.

# Temperature sweep only
python -m src.run_experiments --group temp --num_samples 5

# Full sweep across all conditions
python -m src.run_experiments --group all --output_dir ./results/sweep_ultrakill

# Supply custom prompts
python -m src.run_experiments --group temp --prompts_file ./data/eval_prompts.jsonl

Available groups: greedy, temp, topk, topp, all.

4. Evaluate with Automated Metrics

Compute FAD (requires reference audio), CLAP prompt adherence, repetition, and diversity for every condition.

python -m src.metrics.evaluate_all \
    --manifest ./results/sweep_ultrakill/manifest.json \
    --reference ./data/ultrakill \
    --output_dir ./results/metrics_ultrakill \
    --device cuda

# Skip FAD (no reference audio) or CLAP (no GPU / slow)
python -m src.metrics.evaluate_all \
    --manifest  ./results/sweep_ultrakill/manifest.json \
    --reference ./data/ultrakill \
    --skip_fad  --output_dir ./results/metrics_ultrakill

Skip FAD or CLAP with --skip_fad / --skip_clap.

5. Human Listening Study

Step 1 — Generate pairwise tasks:

python -m src.human_study.generate_pairs \
    --manifest ./results/sweep_ultrakill/manifest.json \
    --conditions greedy temp_1.0 temp_1.5 topk_250 topp_0.95 \
    --pairs_per_prompt 3 \
    --output_file ./results/human_study/pairs.json \
    --project_root .   # makes audio paths relative for web serving

Step 2 — Serve the web interface:

cd /path/to/MusicGen
python -m http.server 8080

Open http://localhost:8080/src/human_study/viewer.html in a browser, load pairs.json, and start rating. Responses can be exported as CSV.

6. Analyze Correlations

Generate a correlation heatmap from the summary metrics:

python -m src.metrics.analysis --metrics_path ./results/metrics_ultrakill/summary.json