ByteDance Seed's DiffusionOPSD: full-trajectory rewards for diffusion RL, 60% less GPU time
andrew_n_carr · x · 2026-08-29
ByteDance's Seed team released DiffusionOPSD (On-Policy Self-Distillation in Diffusion Models), tackling a core pain point of RL for diffusion models: endpoint rewards only arrive after the full image is decoded, leaving intermediate denoising steps unsupervised.
Method:
- A frozen behavior policy generates trajectories supplying low-noise query states and clean-output anchors;
- Normalized reward ascent/descent is converted into bounded positive and negative targets within a controlled clean-output radius;
- The trainable policy fits detached targets under a finite budget, then EMA refreshes the behavior policy for the next iteration.
Results: Extensive ablations on SD3.5-M and step-distilled Z-Image-Turbo across ten evaluators; best final held-out scores in reward-matched settings with roughly 60% fewer training GPU-hours than DiffusionNFT. Paper and code are public.
More from Research
- DeepPhylo infers phylogenetic trees without unrealistic evolutionary assumptions — anshulkundaje · 2026-10-15
- Wander preprint explores graph foundation models via shared objective and random-walk interface — mmbronstein · 2026-10-15
- CHI Reviewer: AI Slop Is Real, but Unusually Rigorous Papers Still Getting Through — IanArawjo · 2026-10-15
- Into the Unknown: CMU benchmark finds 114 of 135 AI-written research papers fail integrity checks — arankomatsuzaki · 2026-10-15
- 8 non-negotiable principles for building AI that actually learns from experience — tokenbender · 2026-10-15
- Benchling launches Science Foundries: multiplayer agent swarms for end-to-end science workflows — nlarusstone · 2026-10-15