Release minimal codebase for MSMD based on FLAME topology#4
Open
EvanSamaa wants to merge 5 commits intoubisoft:mainfrom
Open
Release minimal codebase for MSMD based on FLAME topology#4EvanSamaa wants to merge 5 commits intoubisoft:mainfrom
EvanSamaa wants to merge 5 commits intoubisoft:mainfrom
Conversation
Updated checkpoint iteration in README and added download section for pretrained model.
Updated section header for pretrained models setup and added download link.
Updated description to clarify authorship and data usage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a clean public release of the MSMD codebase, restructured from the original research code. The original was adapted from [DiffPoseTalk](https://github.com/DiffPoseTalk/DiffPoseTalk) and the internal
ubisoft-laforge-msmdmain branch. This release uses the FLAME topology with non-proprietary data, removes the dependency on the proprietary SEREP encoder, and makes the core training and inference pipelines runnable out of the box.What's changed
msmd/models/,msmd/data/,msmd/utils/,msmd/options/, withdata/andpretrained_models/as top-level directoriespretrained_models/flame/sys.path.appendhacks: the package is now properly installable and importablebenchmark_Inference_DPT_runtime.py: was importing fromrefactor_train_dpt(nonexistent); corrected tomsmd.data.dataset_factoryexpressive-speech2faceimports from inference filesinference.py: runs on a single audio file with no dataset required; outputs motion coefficients as.npyor.jsontrain_diffusion.py,train_se.py): removed unused visualization imports, updated experiment output dirs toexperiments/scripts/:train_msmd.sh,train_dpt.sh,train_se.shupdated with relative paths and plain bash (SLURM directives retained but commented out)README.md: covers installation, FLAME setup, dataset structure, training, and inferencerequirements.txtFiles not included in this release
expressive-speech2face/— spiral autoencoder, used only for visualizationjam_mesh/,annotation_scripts/— preprocessing onlyMSMD_venv/Testing
python -c "from msmd.models.diff_talking_head import get_difftalkinghead_model"passespython -c "from msmd.data.dataset_factory import create_datasets"passespython inference.py --helpprints usagepython train_diffusion.py --helpprints usage