iADD corrects DDPO theory: latter-timestep-only updates can harm diversity

Ashok Prasad Neupane · hf · 2026-10-07

This paper revisits RL post-training of diffusion models (e.g., DDPO), which trades off diversity and quality for reward alignment.

It mathematically shows that updating only the latter diffusion timesteps can harm diversity, contradicting a prior work's conclusion.
The authors propose an incremental Feynman-Kac training scheme achieving the best-yet alignment-diversity tradeoff.
Experiments across three tasks with strong ablations validate gains in both alignment and diversity.

More from Multimodal