🎯 BalancedDPO: Adaptive Multi-Metric Alignment

Align text-to-image diffusion models to many preference metrics at once — with a majority-vote consensus inside DPO.

Dipesh Tamboli · Souradip Chakraborty · Aditya Malusare · Biplab Banerjee · Amrit Singh Bedi · Vaneet Aggarwal

Transactions on Machine Learning Research (TMLR), 2026

Single-metric alignment tends to win on one axis while failing another (e.g. aesthetics vs. prompt-alignment).
BalancedDPO combines multiple preference metrics and stays strong across all of them.

---

💡 TL;DR

Aligning text-to-image diffusion models with human preference usually means optimizing a single reward — aesthetics, or a CLIP/semantic score, or a human-preference model. That over-fits whichever metric you pick and neglects the rest.

BalancedDPO aligns to several preference metrics at once inside Direct Preference Optimization (DPO): instead of averaging rewards (where scale differences let one metric dominate), it takes a majority vote among diverse preference scorers to decide the preferred image, and feeds that consensus into DPO with dynamically updated reference models for more stable training — all while keeping the standard DPO pipeline.

✨ Key ideas

• The problem — real preferences involve multiple, sometimes conflicting metrics (semantic consistency, aesthetics, human preference). Single-metric or scalarized (weighted-sum) rewards bias the model toward one criterion and let reward scale differences dominate.
• Majority-vote consensus — aggregate the agreement across diverse scorers in the preference space rather than mixing raw reward values, sidestepping reward-scale conflicts.
• Consensus-in-the-loop DPO — the vote is fed directly into the DPO objective with dynamic reference-model updates, giving stabler gradient directions across heterogeneous metrics.
• Simple & scalable — no separate reward model to train or query at inference; it stays a drop-in DPO pipeline.

🔍 How it works

For each prompt, a set of preference scorers S₁ … Sₙ (e.g. human-preference, CLIP, aesthetic) independently vote for the better of two candidate images. The majority vote becomes the preference label, which is optimized with the DPO objective under dynamically updated reference models. Because the decision happens in preference space, metrics with very different numeric scales can no longer drown one another out.

📈 Results

BalancedDPO reaches state-of-the-art preference win rates on Pick-a-Pic, PartiPrompt, and HPD, and the gains hold across backbones — Stable Diffusion 1.5, SD 2.1, and SDXL.

Average win-rate improvement over DiffusionDPO:

Pick-a-Pic	PartiPrompt	HPD
+15.0%	+7.1%	+10.3%

Win rate (%) — BalancedDPO vs. SD 1.5 (SD 1.5 backbone; >50% means BalancedDPO is preferred more often)

Metric	Pick-a-Pic	PartiPrompt	HPD
HPS	85.2	70.1	72.7
CLIP	78.5	65.2	72.7
PickScore	64.3	56.3	60.7
Aesthetic	69.4	62.2	63.3

Ablations confirm that both the majority-vote aggregation and the dynamic reference updating drive the gains.

🖼️ Qualitative comparison

Same prompt, same seed. Top: "person riding a shark" — BalancedDPO renders the person that both baselines drop. Bottom: for a safe-for-work prompt, the baselines produce NSFW content (masked by the authors) while BalancedDPO stays on-prompt and SFW.

🔗 Links

• 📄 Paper: arXiv:2503.12575 · OpenReview (TMLR)
• 🌐 Project page: dipeshtamboli.github.io/balancedDPO

🚧 Code

Training and evaluation code is being cleaned up for release. ⭐ Star / watch this repo to be notified when it lands. In the meantime, the paper and project page describe the full method and experiments.

📚 Citation

If you find BalancedDPO useful, please cite:

@article{tamboli2026balanceddpo,
  title   = {BalancedDPO: Adaptive Multi-Metric Alignment},
  author  = {Tamboli, Dipesh and Chakraborty, Souradip and Malusare, Aditya and
             Banerjee, Biplab and Bedi, Amrit Singh and Aggarwal, Vaneet},
  journal = {Transactions on Machine Learning Research},
  year    = {2026},
  url     = {https://openreview.net/forum?id=8HRID5VLQw}
}

📄 License

Released under the MIT License.