Scaling Reinforcement Learning for Diffusion Models via Velocity Matching
Jaemoo Choi, Wei Guo, Yuchen Zhu, Arash Vahdat, Molei Tao, Julius Berner, Yongxin Chen
cs.CV, cs.LG
2026-08-25
RVM fine-tunes diffusion models by reward-weighted velocity regression, no likelihoods. Wan2.1-1.3B reaches VBench 84.13 at 525 GPU-h.
Reward fine-tuning for diffusion models still arrives wearing language-model clothes. PPO and GRPO need a likelihood. Autoregressive models have one, token by token. Diffusion models do not: the marginal probability of a finished sample is intractable. Trajectory methods fake it with per-step Gaussian transitions, which forces SDE sampling and storage of the whole denoising path, then dumps a single terminal reward onto a chain of random intermediate states. ELBO methods drop the path and score the endpoint, but they still estimate a policy ratio.
Unroll an ELBO policy gradient and the parameter update is already reward-weighted velocity regression. Georgia Tech and NVIDIA cut the likelihood layer and write the update in the representation the model was trained in: reward-based velocity matching (RVM).
Flow matching already regresses a velocity field. Clean sample x0 and noise ε interpolate to xt; the target is v = ε − x0. RVM does four things:
No policy ratio, no clipping, no stored trajectory. Video runs use a 16-step deterministic ODE, no CFG, and default β=0, so the update is pure reward-weighted regression. Wan2.1-T2V-1.3B is LoRA-tuned at rank 128 for 90 epochs on 8×H100.
RAM and DiffusionNFT sit inside the same gradient. RAM is on-policy RVM with a reference anchor and β=1. DiffusionNFT uses an EMA anchor and maps a [0, 1] reward onto a signed weight. The algebra of the loss barely moves the numbers. Reward shaping and the anchor do.
Video makes that concrete. VideoAlign and HPSv3 will happily reward a clean, nearly frozen clip. The fix is a dynamic-tracking (DT) reward: RAFT optical flow on the fastest 5% of pixels, compared against a resolution-aware threshold. The I2V variant subtracts median flow and only scores mid-range foreground motion, so a drifting background earns nothing. Default mix: text alignment 1.5, motion quality 1.0, HPSv3 0.2, DT 0.7.
Wan2.1-T2V-1.3B on VBench, one video per prompt:
| Method | Dynamic Degree | Overall | GPU-h (8×H100) |
| Base, no CFG | 54.17 | 76.02 | n/a |
| Base, CFG=5 | 65.28 | 83.10 | n/a |
| FlowGRPO | 55.56 | 75.91 | 1,159 |
| DanceGRPO† | 58.33 | 79.46 | 6,171 |
| TaRoS-72B† | 58.33 | 80.56 | 6,171 |
| DiffusionNFT | 63.89 | 82.95 | n/a |
| RAM | 61.11 | 81.95 | n/a |
| RVM | 75.00 | 84.13 | 525 |
†DanceGRPO / TaRoS numbers are cited under a different protocol (CFG, 50-step SDE, five videos per prompt). Against FlowGRPO the recipe is 2.2× cheaper; against the DanceGRPO / TaRoS protocol, 11.8×. The gap is a 16-step CFG-free ODE plus a single noised state per update.
Drop DT and dynamic degree collapses to 5.56, Overall 80.39. Add it and the scores jump to 75.00 / 84.13. Motion is a reward problem.
On SkyReels-I2V, FlowGRPO drags Overall from 81.68 to 77.97. RVM reaches 86.27 with dynamic degree 72.36 (base 64.63). NFT 86.16 and RAM 86.61 sit in the same cluster. On SD3.5-M OCR, RVM hits OCR 0.95 and PickScore 23.03, level with or slightly above the ELBO method PEPG (0.94 / 22.93).
Few-step sampling holds up. The CFG base falls from 83.10 at 16 steps to 73.62 at 5 and 64.39 at 4. CFG-free RVM loses 0.43 and 1.83; four-step RVM still matches the 16-step CFG base. Unanchored training wins on Total. Anchoring to a CFG=5 reference lifts Semantic from 76.29 to 78.48. RVM Semantic 76.28 still trails the CFG base at 80.64; Overall is carried by quality and motion.
Video diffusion post-training does not need trajectory likelihoods. The velocity field is the native interface, and reward-weighted regression plugs into a deterministic ODE. RVM, RAM, and DiffusionNFT land in a tight band, so a new loss functional is a weak use of time. Reward design is not. If motion is not scored, preference models will freeze the clip.
The claim is a change of coordinates: pose diffusion reward fine-tuning in velocity space. RVM does not add a sampler or a backbone. The increment is the recipe and the motion reward.
There is no standalone limitations section. The 11.8× cost figure mixes iteration count and sampler settings with the algorithm, so it is not a same-protocol speedup. Main tables use one video per prompt and report no variance. Default video runs set β=0, which sits awkwardly next to the "anchored unification" story. The fixed-point remark is honest: the learned field need not equal the conditional velocity of its own endpoint distribution. DT depends on RAFT and hand-set thresholds; the I2V thresholds come from human labels on earlier fine-tuning runs. The only named follow-up is autoregressive video, with no experiment.