ByteDance Seed's DiffusionOPSD: full-trajectory rewards for diffusion RL, 60% less GPU time
andrew_n_carr · x · 2026-08-29
ByteDance's Seed team released DiffusionOPSD (On-Policy Self-Distillation in Diffusion Models), tackling a core pain point of RL for diffusion models: endpoint rewards only arrive after the full image is decoded, leaving intermediate denoising steps unsupervised.
Method:
- A frozen behavior policy generates trajectories supplying low-noise query states and clean-output anchors;
- Normalized reward ascent/descent is converted into bounded positive and negative targets within a controlled clean-output radius;
- The trainable policy fits detached targets under a finite budget, then EMA refreshes the behavior policy for the next iteration.
Results: Extensive ablations on SD3.5-M and step-distilled Z-Image-Turbo across ten evaluators; best final held-out scores in reward-matched settings with roughly 60% fewer training GPU-hours than DiffusionNFT. Paper and code are public.
More from Research
- Pathway CSO pitches BDH as post-Transformer architecture uniting memory, reasoning, adaptation — schwentker · 2026-10-15
- EmbodiedEvo tops RoboDojo at 38.95% success by letting robots inherit failed practice — thetripathi58 · 2026-10-15
- 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
- Google's HARNESS² evolves agent harnesses without verifiers, up to +11.5 points on WorkBuddy-Bench — dair_ai · 2026-10-15
- Subbarao Kambhampati to dissect the 'mythos of LLM thinking tokens' at UMN — rao2z · 2026-10-15
- How 3D Gaussian Splatting gives robots spatial memory of what's out of view — TheTuringPost · 2026-10-15