Think Before You Score: Thinking Reward Model for Visual Generation
Xuehai Bai, Zhenchen Tang, Yang Shi, Dianyi Wang, Tengfei Liu, Wanshun Su, Xuanyu Zhu, Ruohui Wang, Haiwen Diao, Haotian Wang, Xiaoling Gu, Yuanxing Zhang
cs.CV
2026-09-29
TRM writes a case-specific rubric before scoring. The 9B model hits 71.2% on GenAI-T2I versus GPT-4.1 at 60.5%, and as an RL reward lifts several image generation and editing models.
Visual RL post-training needs a reward that turns human preference into a number a generator can climb. Most visual reward models map the task condition and a candidate image straight to a scalar or a pairwise vote. ImageReward, UnifiedReward, and EditScore all work that way. Which objects this prompt actually asked for, and which parts of an edit must stay still, stay implicit inside that mapping.
A few recent models reason before they score. RationalRewards and EditScore write an analysis first. Visual judgment still changes with the instance. Generation cares about object relations, anatomy, and rendered text; editing also cares about identity and accidental changes. A fixed checklist misses items. Direct scoring never says what it looked at. The missing step is deciding, for this case, what to evaluate.
TRM scores only after it writes a plan. The pipeline is R → J → H → s: emit a case-adaptive rubric of atomic checks, answer each Yes or No, summarize by dimension, then output a pointwise score. Generation and editing share Prompt Alignment and Technical Quality. The third dimension is Aesthetics for generation and Source Consistency for editing, meaning whether unrequested content survived. The three buckets stay fixed; the actual checks are written from the current condition and the candidate, like listing mark-scheme items for this question before grading.
Training has two stages. About 20K generation cases and 28K editing cases go through a two-stage human-AI annotation: Gemini drafts rubrics, experts review coverage, atomicity, and verifiability; the same rubrics then drive judgments and scores, with expert calibration. SFT mixes three formats in one run: rubric only, score given a rubric, and the full trace. The base is Qwen3.5-9B with LoRA; the vision tower stays frozen.
After SFT, about 4,000 preference pairs drive RL. A Bradley-Terry objective on pointwise scores keeps paying for a larger gap after the order is already correct, so scores drift to the ends of the range. PD-GRPO samples 8 independent pointwise evaluations for the winner and 8 for the loser. A winner rollout gets reward 1 only if its score beats the loser-group mean by margin m; losers are symmetric. Once the margin holds, widening the gap pays nothing extra. Policy updates pool the 16 responses into one group-relative advantage. At inference the model still scores one image at a time. Editing uses m=0; generation uses m=0.05 on normalized scores.
Preference accuracy on image-generation reward benchmarks, excluding predicted ties:
| Method | GenAI-T2I | MMRB2-T2I |
| TRM (RL) 9B | 71.2 | 67.9 |
| TRM (SFT) 9B | 70.1 | 65.8 |
| Qwen3.5-9B pointwise baseline | 58.9 | 59.4 |
| GPT-4.1 | 60.5 | 65.8 |
| Gemini 3 Pro | 73.1 | 74.4 |
| HPSv3 7B | 70.8 | 60.2 |
The 9B TRM beats every open specialized reward model they report, and beats GPT-4.1. Gemini 3 Pro still leads. PD-GRPO adds 1.1 and 2.1 points over SFT. The pointwise Qwen3.5-9B baseline ties on 40.9% and 56.6% of pairs; TRM cuts that to 12.2% and 20.7%. With ties scored as 0.5, TRM (RL) is 68.4% and 63.9%.
On editing, TRM (RL) reaches 0.786 / 0.674 / 0.773 on EditScore-ERB (instruction following / visual consistency / overall) against 0.440 / 0.308 / 0.401 for the pointwise baseline and 0.635 / 0.586 / 0.703 for 72B EditScore. MMRB2 rises from 30.9% to 58.2%. Gemini 3.1 Pro remains clearly ahead.
Used as the sole Flow-GRPO reward, TRM lifts BAGEL from 0.86 to 0.89 on GenEval and 75.62 to 81.43 on TIIF-Long, and FLUX.1-dev from 70.84 to 77.60 on TIIF-Short. On editing, BAGEL ImgEdit goes from 3.37 to 3.91; SenseNova-U1.5 from 4.30 to 4.52. Against AlphaGRPO on the same BAGEL, TIIF-Short is 80.68 vs 77.70. Against 72B EditScore on the same SenseNova, 9B TRM is 4.52 vs 4.51 on ImgEdit; at step 300, within-group reward std falls 14.9% with TRM and 50.5% with EditScore-72B.
For visual RL, this is a usable open 9B pointwise reward: serving is a single-image score, training still sees pairwise preferences, so cost grows with group size rather than its square. The rubric sits in the output, which is easier to debug than a bare scalar. Downstream it matches 72B EditScore at about one-eighth the parameters, which matters when the reward is queried on every rollout.
The idea is incremental. Rubric-guided reasoning rewards already exist. TRM makes case-adaptive rubric generation a shared procedure for generation and editing, and PD-GRPO stops Bradley-Terry from polarizing scores. The overview figure sketches video and world models; those tasks are not in the experiments.
The paper has no limitations section. Experiments stop at image generation and editing. Gemini 3 Pro and Gemini 3.1 Pro still lead most reward-modeling numbers. Labels come from a Gemini teacher plus expert calibration, so scores may inherit teacher taste; the authors already found a single teacher too weakly separated on generation and added other evaluators. The vision tower is frozen throughout, so fused fingers can appear in the rubric while the visual encoder itself is never updated by reward training.
Headline generation accuracies drop tied predictions. The Qwen3.5-9B pointwise baseline ties on over half of MMRB2-T2I pairs; reporting only strict-order pairs inflates the number. Downstream RL never compares against GPT-4.1 or Gemini as the reward, so the claim that a better reward model yields a better generator is only shown among open rewards. SenseNova-U1.5 shares authors' affiliation; that gain should be read with a discount. They reduce MLLM reward latency with async scoring and prefix cache, but never report wall-clock against a regression reward model.