SynMotion

Official inference code for SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation.

• Project page: https://lucaria-academy.github.io/SynMotion/
• Paper: https://arxiv.org/abs/2506.23690
• SynMotion weights: https://huggingface.co/Shuaishuai0219/SynMotion
• Base model: https://huggingface.co/tencent/HunyuanVideo-I2V

SynMotion learns a customized motion from a small set of examples and transfers it to new subjects. Each motion checkpoint contains two branches:

<synmotion_weights>/
  dance/
    dual_embedding.pth
    visual_adapter.safetensors
  doublewave/
    dual_embedding.pth
    visual_adapter.safetensors
  liftarm/
    dual_embedding.pth
    visual_adapter.safetensors
  pray/
    dual_embedding.pth
    visual_adapter.safetensors
  salute/
    dual_embedding.pth
    visual_adapter.safetensors
  squat/
    dual_embedding.pth
    visual_adapter.safetensors
  throw/
    dual_embedding.pth
    visual_adapter.safetensors
  wave/
    dual_embedding.pth
    visual_adapter.safetensors

dual_embedding.pth stores the semantic dual-embedding branch, and visual_adapter.safetensors stores the visual motion adapter branch.

News

• 2026-07-01: Public inference entrypoints and checkpoint layout support are prepared for release.
• 2026: SynMotion is accepted by CVPR 2026.

Requirements

The code is tested on Linux with NVIDIA GPUs. HunyuanVideo-I2V is a large video model; an 80 GB GPU is recommended for 720p generation. Lower resolutions such as 540p can reduce memory usage.

Create the environment:

conda create -n synmotion python=3.11.9
conda activate synmotion
conda install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.4 -c pytorch -c nvidia
python -m pip install -r requirements.txt
python -m pip install ninja
python -m pip install flash-attn==2.6.3 --no-build-isolation

For multi-GPU sequence-parallel inference, install xDiT:

python -m pip install xfuser==0.4.0

Model Weights

Download the HunyuanVideo-I2V base model and SynMotion motion weights:

huggingface-cli download tencent/HunyuanVideo-I2V --local-dir ./ckpts/HunyuanVideo-I2V
huggingface-cli download Shuaishuai0219/SynMotion --local-dir ./ckpts/SynMotion

The base model root passed to --model-base should contain the HunyuanVideo-I2V folders used by this code:

ckpts/HunyuanVideo-I2V/
  hunyuan-video-i2v-720p/
    transformers/
    vae/
  text_encoder_i2v/
  text_encoder_2/

The SynMotion root passed to --synmotion-checkpoint-root should contain one subfolder per motion. The expected motion folder names are dance, doublewave, liftarm, pray, salute, squat, throw, and wave.

Single-Image Inference

Run one image-to-video sample with a named motion:

python sample_synmotion.py \
  --model-base ./ckpts/HunyuanVideo-I2V \
  --synmotion-checkpoint-root ./ckpts/SynMotion \
  --motion-name dance \
  --prompt "A penguin dances." \
  --i2v-image-path ./examples/penguin.png \
  --i2v-resolution 540p \
  --video-size 540 960 \
  --video-length 129 \
  --infer-steps 50 \
  --seed 0 \
  --save-path ./results/dance

You can also point directly to a single motion folder:

python sample_synmotion.py \
  --model-base ./ckpts/HunyuanVideo-I2V \
  --root-checkpoint-path ./ckpts/SynMotion/dance \
  --prompt "A penguin dances." \
  --i2v-image-path ./examples/penguin.png \
  --i2v-resolution 540p \
  --video-size 540 960 \
  --save-path ./results/dance

Batch Inference

For a directory of images, use {subject} in the prompt. The subject name is inferred from each image filename.

python sample_synmotion_batch.py \
  --model-base ./ckpts/HunyuanVideo-I2V \
  --root-checkpoint-path ./ckpts/SynMotion/wave \
  --prompt "A {subject} waves." \
  --i2v-image-path ./examples/animals \
  --i2v-resolution 540p \
  --video-size 540 960 \
  --seed 0 \
  --save-path ./results/wave_batch

The legacy entrypoints remain available for compatibility:

• sample_video.py forwards to sample_synmotion.py
• sample_image2video_lora_inverse.py forwards to sample_synmotion.py
• sample_image2video_lora_inverse_batch.py forwards to sample_synmotion_batch.py

Useful Arguments

Argument	Description
--model-base	Local HunyuanVideo-I2V model root.
--synmotion-checkpoint-root	Local SynMotion weight root containing motion subfolders.
--motion-name	Motion subfolder name under --synmotion-checkpoint-root.
--root-checkpoint-path	Direct path to one motion folder.
--dual-embedding-path	Direct path to dual_embedding.pth.
--visual-adapter-path	Direct path to visual_adapter.safetensors.
--prompt	Generation prompt. Batch mode supports {subject} or {}.
--i2v-image-path	Reference image path, or an image directory in batch mode.
--i2v-resolution	Resolution bucket: 360p, 540p, or 720p.
--video-size	Output height and width.
--video-length	Number of frames. It should satisfy video_length - 1 being divisible by 4.
--infer-steps	Number of denoising steps.
--save-path	Output directory.

Troubleshooting

• If the code reports missing dual_embedding.pth or visual_adapter.safetensors, check that the local folder contains one subfolder per motion and that --motion-name matches the folder name exactly.
• If the code reports missing HunyuanVideo weights, pass --model-base to the directory containing hunyuan-video-i2v-720p, text_encoder_i2v, and text_encoder_2.
• If the generated video consumes too much memory, try --i2v-resolution 540p --video-size 540 960, fewer frames, or sequence-parallel inference.
• The scripts do not download weights automatically. This keeps inference deterministic and avoids partial checkpoint downloads during runs.

Acknowledgements

This implementation builds on the open-source HunyuanVideo-I2V codebase and uses components from Diffusers, Transformers, PyTorch, and Safetensors. Please also follow the licenses and model terms of the base model and dependencies.

Citation

If you find this project useful, please cite:

@inproceedings{tan2026synmotion,
  title={SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation},
  author={Tan, Shuai and Gong, Biao and Wei, Yujie and Zhang, Shiwei and Liu, Zhuoxin and Ma, Ke and Wang, Yan and Zheng, Kecheng and Zhu, Xing and Shen, Yujun and Zhao, Hengshuang},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2026}
}