RL for Diffusion Models: A Deep Dive on Why Rewards Beat DPO Pairs

kastnerkyle · x · 2026-11-03

A 27-minute blog post (Part 1 of a series) explains why diffusion models need RL beyond preference optimization.

DPO's weaknesses: the model drifts from its pretraining distribution (requiring SFT on winners or methods like ThermoDPO), and winner/loser pairs only say "this image is better"—they don't tell the model what to fix. A "four cups beats three cups" pair won't teach the model that its mistake was counting.
Rewards localize errors: detectors can count cups, OCR can check sign text, separate models can score aesthetics or hand anatomy—anything scoreable can be optimized.
Assuming a good reward model exists, the post covers how to update the generator to achieve higher reward more often.

More from Multimodal