DDRL keeps diffusion loss during RL to fix reward hacking in video models
gowthamis posted a multi-tweet walkthrough of the arXiv paper "Data-regularized Reinforcement Learning for Diffusion Models at Scale" (Haotian Ye, James Zou, et al.). The paper identifies reward hacking in RL fine-tuning of video diffusion models—FlowGRPO's KL value stabilizes even with larger KL penalties, yet generated videos still show noisy textures, because the penalty only compares models on states visited by the current policy and loses force once the policy drifts. DDRL keeps the diffusion (denoising) loss during RL training, which fixes the issue.
Confirmed
- Setup: full finetune on Cosmos2.5 pre-RL checkpoints (2B/14B), 93 frames at 720p, 1,044 test prompts
- On 2B text-to-video, DDRL achieves lower reward yet wins human preference by 10.5 percentage points (VideoAli… metrics)
- Corresponding results also reported on 2B text-to-image
- Ablations: removing the separate 20,000-step SFT stage and running DDRL directly from the pretrained checkpoint yields reward and diffusion loss comparable to the SFT→DDRL pipeline, so SFT can be skipped with large gains in data efficiency
Why it matters
- It explains why KL penalties fail at scale in diffusion RL and why metrics can stabilize while visual quality degrades
- Retaining the diffusion loss is a simple, effective regularizer that also simplifies training by eliminating the standalone SFT stage
2026-10-15 ~ 2026-10-15 · 5 related posts
Primary sources
- DDRL anchors diffusion RL with data: 1M+ GPU hours show it beats reward hacking in video generation — gowthami_s ·
- DDRL scores lower reward than DanceGRPO yet wins human preference by 10.5 pts — gowthami_s ·
- DDRL ablations show SFT stage can be skipped, matching rewards straight from pretraining — gowthami_s ·
- [source] DDRL anchors diffusion RL with data: 1M+ GPU hours show it beats reward hacking in video generation — gowthami_s · 2026-10-15
- KL penalty fails in video RL: FlowGRPO stabilizes KL yet outputs stay noisy — gowthami_s · 2026-10-15
- [source] DDRL scores lower reward than DanceGRPO yet wins human preference by 10.5 pts — gowthami_s · 2026-10-15
- [source] DDRL ablations show SFT stage can be skipped, matching rewards straight from pretraining — gowthami_s · 2026-10-15
- DDRL ablation: skipping SFT and running RL straight from pretraining works — gowthami_s · 2026-10-15