Paired finite differences make text-to-image RL 19× faster than Flow-GRPO

Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models

David McAllister, Miika Aittala, Tero Karras, Janne Hellsten, Angjoo Kanazawa, Timo Aila, Samuli Laine

cs.CV, cs.AI, cs.LG, cs.NE, stat.ML

2026-03-13

FDFO treats the sample as one action and updates the flow with a paired reward gap. On SD 3.5 Medium, combined reward arrives ~19× sooner than Flow-GRPO, with no grid artifacts.

What problem this solves

Flow matching turns text-to-image generation into a velocity field. The network predicts each step, and an ODE carries noise to an image. Pre-training pins that field to the data. Post-training wants softer targets: better looking pictures, steadier composition, more expressive light. Those do not make a loss. Proxy rewards stand in, such as a human-preference score or a check that rendered text is right.

DDPO, Flow-GRPO, and DanceGRPO cast stochastic sampling as a Markov decision process. Each denoising step is a Gaussian action. If the finished trajectory scores above its group, velocity is pulled toward the random kick taken on that step. The average update points at higher reward, but close to half of the step-level updates can point the other way. Most of the magnitude is reward-neutral noise. Axes the reward does not see drift, and style wanders. Given long enough, that drift becomes damage and caps how far fine-tuning can go. Extended Flow-GRPO runs grow grid artifacts that later fade.

Method

The work is from NVIDIA and UC Berkeley. Finite difference flow optimization (FDFO) treats the whole sample as one action. Two trajectories leave the same noise, with only a little stochasticity, so detail changes and layout mostly stays. The image gap Δx times the reward gap ΔR points from the worse image to the better one. Every timestep on both trajectories bends its velocity that way. An MDP reinforces whatever kick a step happened to take. FDFO writes the same endpoint comparison into every step.

This assumes diffusion flows barely rotate. Add a little signal on a half-noised image and later denoising keeps the difference in the final picture, the way a dab of color on a rough sketch grows into an object. Strictly, the Jacobian of the map that remains should be positive semi-definite. Optimal transport has that property. Diffusion flows only resemble it numerically, and counterexamples exist.

Plain Euler-Maruyama queries the network at an inconsistent time and noise level, and some steps inject too much noise. FDFO uses the EDM stochastic sampler: follow the ODE past the target time, then add fresh noise to land on the next step. Default strength is γ = 0.0025 on every step, about the fraction of noise redrawn there. More complicated schedules did not help reliably. ΔR scales with the length of Δx, so the raw product counts that length twice. Training normalizes Δx by its RMS norm, with 1e-6 in the denominator.

Compute matches Flow-GRPO. The base model is Stable Diffusion 3.5 Medium. Only LoRA trains, low-rank layers on a frozen network. Resolution is 512×512, with 40 steps in training and at inference. Each epoch rolls out 432 pairs, stores all 40 steps, and splits them into 4 batches of 8640 samples updated with AdamW. Several updates reuse the same rollouts, so SPO clipping, in the same family as PPO-Clip, downweights velocities that have already moved too far. The main runs turn off KL regularization and classifier-free guidance. CFG amplifies the gap between conditional and unconditional velocity and can hide algorithm differences. Model forwards and reward calls per epoch match Flow-GRPO.

Prompts come from the Pick-a-Pic training set. PickScore targets human preference. A second reward asks the vision-language model Qwen2.5-VL-7B-Instruct whether the image matches the caption, then takes 100 times a sigmoid of the Yes logit minus the No logit. The combined reward is PickScore plus that VLM score divided by 10.

Results

At equal epoch cost both methods raise PickScore. FDFO gets higher, sooner. Flow-GRPO reaches 23.43 at 1000 epochs (2000 optimizer steps), in line with its own paper. VLM alignment is harder, and the gap widens. The combined reward shows the same pattern.

OneIG-Bench measures prompt alignment and diversity. HPSv2 measures human preference. Both are scored after the fact on saved checkpoints. PickScore barely moves alignment. The VLM reward targets it. The combined reward is steadier, and with FDFO it also leads on preference. All three rewards cut diversity, PickScore fastest and the VLM slowest. At a matched reward, the two algorithms lose diversity at about the same rate.

Ablations use the combined reward. Entries are OneIG alignment.

Setup200 epochsBest
Flow-GRPO71.0679.56
FDFO sampler76.7482.94
Then finite differences82.1483.07
Shared initial noise81.1183.95
Full FDFO82.1384.25
One trajectory deterministic79.7684.02
PPO clip instead of SPO81.9983.91
No Δx normalization79.7480.78
True reward gradient81.4783.73
Gradient through SDE steps66.6968.78

After the new sampler and the switch to finite differences, the 200-epoch score moves from 71.06 to 82.14, and the best score from 79.56 to 83.07. Sharing the initial noise then drops the 200-epoch number to 81.11 while the best rises to 83.95. Full FDFO peaks at 84.25. A deterministic partner trajectory reaches 84.02, but only 79.76 at 200 epochs. PPO matches SPO only with an interval schedule on γ, the one setting where a constant γ was clearly worse. Without normalization the best score stalls at 80.78. A true reward gradient is no better. Pushing that gradient back through the SDE steps falls to 66.69 at 200 epochs, under Flow-GRPO's 71.06.

Wall clock is NVIDIA H200 hours. To cross the same combined reward, 40-step FDFO takes about 29.3 hours against about 562.5 for Flow-GRPO, stated as 19×. The 10-step setup is about 17.3 against 89.6 hours, stated as 5×. The public Flow-GRPO code has extra overhead. A zero-overhead estimate still favors FDFO. That curve's factor is not stated.

Flow-GRPO drifts in style before rewards match. Hand-picked frames at epoch 830 show heavy grids that later fade, so the combined reward penalizes them only weakly. FDFO at the reward-matched epoch 70 does not show them, including after equally long training.

One author spent about 4 hours on 3,200 on-policy pairs from HPDv2 prompts, with no reward model and no offline preferences, across 50 epochs. Figure 10 is qualitative. The paper estimates that the same 4 hours would not finish a Flow-GRPO-style run.

CFG weights 1, 2, and 4.5 were tried. Before post-training, the base model without CFG is muddy and off prompt. w = 4.5 lifts quality and alignment while cutting diversity. After post-training, skipping CFG keeps more detail. Turning it on mostly reduces variety, adds a high-contrast look, and costs more at inference. Whether to keep it has no numeric answer.

Why it matters

A Flow-GRPO post-training loop can swap in FDFO without more model forwards or reward calls per epoch. At 40 steps the same combined reward arrives about 19× sooner, with fewer grid-like side effects, so fine-tuning can run longer. The reward need not be differentiable: one human comparison is enough for ΔR. When it is differentiable, the true gradient still does not win, so a VLM does not have to sit inside the generator graph. The base model is still a LoRA on SD 3.5 Medium. If the proxy score is wrong, a cleaner update chases that wrong score harder.

Limitations

The main runs disable KL and CFG so regularization does not hide the algorithm gap. Production post-training usually keeps both. KL is compatible, with results in the appendix, and the main text does not say how much speedup remains once it is on. Keeping diversity is left open. A VLM term aimed straight at diversity is the suggested follow-up.

Sending an endpoint difference back to intermediate steps is argued in expectation. Counterexamples exist. γ = 0.0025 and the claim that the flow barely rotates were not rechecked on other schedules or resolutions.

The numeric comparison is basically Flow-GRPO alone. DanceGRPO, DDPO, Diffusion-DPO, ReFL, and DRaFT have no matched scores. The 19× figure includes implementation overhead. The human study is one labeler, with no OneIG or HPSv2. The grid images are hand-picked bad frames. The only model is SD 3.5 Medium at 512 with LoRA. Whether the gap survives another reward or another base model is untested.

Terms

Source

What people are saying

Related papers

All paper explainers