Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion

Anle Ke¹   ·   Xu Zhang¹   ·   Tong Chen¹   ·   Ming Lu¹  ·   Chao Zhou²  ·   Jiawen Gu²  ·   Zhan Ma¹   

¹ Nanjing University   ²Kuaishou Technology 

🌐 Project Page   📃 Paper  

📖 Table Of Contents

• ✨ Visual Results
• ⏳ Train
• 😀 Inference
• 🌊 TODO
• ❤ Acknowledgement
• 🙇‍ Citation

⚙️ Environment Setup

- conda create -n ResULIC python=3.10
- conda activate ResULIC
- pip install -r requirements.txt

✨ Visual Results

⏳ Train

Note: The numbers in the yaml filenames (e.g., 1_1_1) represent $\lambda_{\text{diffusion}}$, $\lambda_{\text{mse}}$, and $\lambda_{\text{bpp}}$ respectively.

Stage 1: Initial Training

1. Download Pretrained Model
   Download the pretrained Stable Diffusion v2.1 model into the ./weight directory:

   wget https://huggingface.co/stabilityai/stable-diffusion-2-1-base/resolve/main/v2-1_512-ema-pruned.ckpt --no-check-certificate -P ./weight
   

2. Modify the configuration file./configs/train_zc_eps.yaml and ./configs/model/stage1/xx.yaml accordingly.

3. Start training.

   bash stage1.sh 
   

Stage 2:

1. Modify the configuration file ./configs/train_stage2.yaml and ./configs/model/stage2/xx.yaml accordingly.

2. Start training.

   bash stage2.py 
   

😀 Inference

1. Download the pre-trained models (We provide 3 low-bitrate models for testing, and other bitrate models can be trained by referring to the training method):

   Bitrate	Link(used config)
   0.03 bpp	ResULIC/configs/model/stage2/1_1_3/cldm_eps_300_ddim.yaml
   0.01 bpp	ResULIC/configs/model/stage2/1_1_12/cldm_eps_400_ddim.yaml
   0.002 bpp	ResULIC/configs/model/stage2/1_1_24/cldm_eps_600_ddim.yaml

Note: add_steps and Q are only used to distinguish the testing image settings from the corresponding model settings. These parameters do not affect the results: add_steps can be aligned with the value in the config file, and Q corresponds to the trained $\lambda$. For the three provided models, the default ddim_steps is 3. If you use your own trained model, it is recommended to set ddim_steps to a divisor of add_steps. For example, when add_steps=500, ddim_steps could be 2, 4, 5...

1. W/o Srr, W/o Pfo.

   CUDA_VISIBLE_DEVICES=2 python inference_win.py \
    --ckpt xx \
    --config /xx/xx.yaml \
    --output xx/ \
    --ddim_steps 3 \
    --ddim_eta 0 \
    --Q x.0 \
    --add_steps x00
   

2. W/ Srr, W/o Pfo.

    CUDA_VISIBLE_DEVICES=2 python inference_res.py \
    --ckpt xx \
    --config /xx/xx.yaml \
    --output xx/ \
    --ddim_steps 3 \
    --ddim_eta 0 \
    --Q x.0 \
    --add_steps x00
   

3. W/ Srr, W/ Pfo.

    CUDA_VISIBLE_DEVICES=2 python inference_res_pfo.py \
    --ckpt xx \
    --config /xx/xx.yaml \
    --output xx/ \
    --ddim_steps 3 \
    --ddim_eta 0 \
    --Q x.0 \
    --add_steps x00
   

🌊 TODO

• Release code
• Release quantitative metrics （👾The quantitative metrics for ResULIC presented in our paper can be found in indicator.）
• Release pretrained models

❤ Acknowledgement

This work is based on ControlNet, ControlNet-XS, DiffEIC, and ELIC, thanks to their invaluable contributions.

🙇‍ Citation

If you find our work useful, please consider citing:

@inproceedings{Ke2025resulic,
               author = {Ke, Anle and Zhang, Xu and Chen, Tong and Lu, Ming and Zhou, Chao and Gu, Jiawen and Ma, Zhan},
               title = {Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion},
               booktitle = {International Conference on Machine Learning},
               year = {2025}
               }