Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Prompts
Shuyue Stella Li, Xiaochuang Han, Yulia Tsvetkov, Luke Zettlemoyer
cs.AI, cs.CV
2026-09-29
VVR scores geometric constraints with pixel-level Python verifiers. RLVVR lifts SD3.5 Medium from 2.8% to 28.3% on VVRBench; counting and position transfer to natural prompts.
Language models can check constraints with code ('use the word X at least three times') and train on that signal. Image generators have no equivalent. Asked for three red circles to the left of two blue squares, they miss counts, colors, or layout, and fail more as constraints stack.
Post-training rewards come from preference models, VLMs that answer questions about the image, and object detectors. Those learned judges err on the exact properties instruction following needs, and policies exploit the errors. This University of Washington work supplies a pixel-level substitute: Python functions over RGB, with no detector, OCR, embedding model, or VLM. They call the framework Verifiable Visual Rewards (VVR).
Each VVR task pairs a natural-language prompt with a formal constraint set. The canvas holds colored geometry: eight colors and three shapes (circle, square, triangle). Constraints fall into five families covering color-shape binding, counts and ratios, regions and layout, relative size, and topology such as contact or containment. The library has 46 types; the main benchmark uses 32.
Tasks are built scene-first:
Visual constraints can be pairwise fine and globally impossible ('A contains B, B contains C, C contains A'), so the generator never starts from free text. Structural complexity C(s) weights each constraint by how many instances it covers, and datasets are sliced on that score.
The verifier is reference-free. Fixed hue thresholds make color masks; connected components become objects; aspect ratio and convexity classify shape; color plus shape bind them to groups. Each constraint returns a hard pass/fail and a partial score in [0, 1]. Benchmark accuracy is the product of hard decisions. Training uses a weighted sum of partial scores times a penalty ψ, so the policy cannot farm one easy constraint and ignore the rest.
RLVVR fine-tunes Stable Diffusion 3.5 Medium with Flow-GRPO, an online policy-gradient method for flow-matching models. Two 100k corpora, decontaminated against the benchmarks: VVR-Easy (complexity at most 20, at most one family) and VVR-Matched (same complexity mix as VVRBench).
Three eval splits. VVRBench: 10,000 tasks, complexity 3 to 48, five bins C1 to C5. Fast: 820 tasks. Challenge: 720 tasks, complexity 45 to 80, plus 14 harder group-level types.
| Model | VVRBench | Challenge |
| GPT-Image-2 | 86.86% | 10.28% |
| GPT-Image-2.5-Sunburst | 84.51% on Fast | 21.39% |
| FLUX.2-dev (best open weights) | 19.15% | not reported |
| pretrained SD3.5 Medium | 2.81% | not reported |
GPT-Image-2 falls from 97.65% in C1 to 65.72% in C5. FLUX.2-dev is at 2.24% in C5. On Challenge, GPT-Image-2.5 drops from 31.67% at complexity 45 to 56 down to 7.92% at 69 to 80; every other model is at most 7.36% (Gemini-3.1-Flash-Lite-Image). Failures sit in cross-group counting and topology: same count 31%, times as many 33%, each contains 34%. Grounding averages about 95%, Topology about 56%. Gemini models sometimes refuse, calling a satisfiable prompt contradictory.
| Training data | VVRBench | C3 / C4 / C5 |
| pretrained SD3.5-M | 2.81% | n/a |
| VVR-Easy | 28.27% | +17.16 / +8.31 / +1.35 vs pretrained |
| VVR-Matched | 46.60% | 45.62% / 38.39% / 21.82% |
Every C3 to C5 task is harder than any Easy training task. Easy closes 79% and 64% of the pretrained-to-Matched gap on partial count and relation scores, but only 55% and 46% on satisfying every count or every relation in a task. Isolated constraints come from easy data; packing many of them into one image comes from large scenes.
Trained only on colored shapes, Easy beats the pretrained model on eight of ten non-VVR metrics: GenEval 0.616 to 0.729, OCR 0.476 to 0.587. On 160 natural prompts outside VVR, humans prefer Easy 71.6% of the time. Against a GenEval2-rewarded model, Easy scores 0.729 vs 0.688 on GenEval, with the lift in position (+0.150), counting (+0.103), and color attribution (+0.025). Mixing Easy with GenEval2 raises all ten non-VVR metrics; humans prefer the mix to GenEval2 alone at 58.6%. Mixing with OCR or a five-reward objective drops native OCR by 0.021 and 0.030, because those mixtures spend fewer prompts on the original target.
Verifier thresholds came from 3,947 human decisions. The released scorer agrees with 481 of 508 audit labels (94.7%, Cohen's κ = 0.89).
Image post-training has lacked a reward a program can grade. VVR ports the IFEval / RLVR idea onto pixels, mints tasks at any complexity, and mixes with GenEval2, OCR, and PickScore. Code, data, and checkpoints are public.
The skill it teaches is hard constraints, not taste. Easy alone lowers HPSv2.1 from 0.300 to 0.294 and aesthetic score from 5.517 to 5.483; a DrawBench vase comes out as a flat unshaded shape. Adding GenEval2 brings shading back. Treat VVR as one term in the recipe, not the whole objective.
Open-weight models sit under 20% on the main set. GPT-Image-2.5 solves 21.39% of Challenge. The benchmark is usable as a probe on its own.
The authors flag the scope: eight colors, three shapes, plain backgrounds, 2D geometry only. No 3D, no editing, no video. RL ran only on SD3.5-M with Flow-GRPO, with no SFT or DPO stage first. The verifier was tested on blur, compression, and merged blobs; whether the same pixel rules survive photographic texture is untested.
The transfer numbers need a discount. GenEval's position and counting slices overlap the skills VVR trains; aesthetics went down. The 71.6% human win rate on 160 natural prompts does not separate 'followed the prompt' from 'looked better.' Challenge uses 20 tasks per complexity integer (720 total). It separates GPT-Image-2.5 from GPT-Image-2 (21.39% vs 10.28%), but the error bars are wide.