Skip to content

Release minimal codebase for MSMD based on FLAME topology#4

Open
EvanSamaa wants to merge 5 commits intoubisoft:mainfrom
EvanSamaa:main
Open

Release minimal codebase for MSMD based on FLAME topology#4
EvanSamaa wants to merge 5 commits intoubisoft:mainfrom
EvanSamaa:main

Conversation

@EvanSamaa
Copy link
Copy Markdown


Summary

This PR introduces a clean public release of the MSMD codebase, restructured from the original research code. The original was adapted from [DiffPoseTalk](https://github.com/DiffPoseTalk/DiffPoseTalk) and the internal ubisoft-laforge-msmd main branch. This release uses the FLAME topology with non-proprietary data, removes the dependency on the proprietary SEREP encoder, and makes the core training and inference pipelines runnable out of the box.


What's changed

  • Replaced SEREP encoder with FLAME: all SEREP calls have been replaced with FLAME equivalents
  • Added pre-trained model weights: enables direct comparison with MSMD in future work
  • Reorganized into a standard ML package layout: msmd/models/, msmd/data/, msmd/utils/, msmd/options/, with data/ and pretrained_models/ as top-level directories
  • Replaced all hardcoded absolute paths with relative paths; FLAME model files now resolve from pretrained_models/flame/
  • Removed all sys.path.append hacks: the package is now properly installable and importable
  • Fixed broken import in benchmark_Inference_DPT_runtime.py: was importing from refactor_train_dpt (nonexistent); corrected to msmd.data.dataset_factory
  • Dropped unused expressive-speech2face imports from inference files
  • Added standalone inference.py: runs on a single audio file with no dataset required; outputs motion coefficients as .npy or .json
  • Cleaned up training scripts (train_diffusion.py, train_se.py): removed unused visualization imports, updated experiment output dirs to experiments/
  • Added scripts/: train_msmd.sh, train_dpt.sh, train_se.sh updated with relative paths and plain bash (SLURM directives retained but commented out)
  • Added README.md: covers installation, FLAME setup, dataset structure, training, and inference
  • Added requirements.txt

Files not included in this release

  • expressive-speech2face/ — spiral autoencoder, used only for visualization
  • jam_mesh/, annotation_scripts/ — preprocessing only
  • Visualization and user study scripts
  • MSMD_venv/

Testing

  • python -c "from msmd.models.diff_talking_head import get_difftalkinghead_model" passes
  • python -c "from msmd.data.dataset_factory import create_datasets" passes
  • python inference.py --help prints usage
  • python train_diffusion.py --help prints usage

Updated checkpoint iteration in README and added download section for pretrained model.
Updated section header for pretrained models setup and added download link.
Updated description to clarify authorship and data usage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant