NTU's V-Rubrics Turns 50K Visual Samples Into 352,938 Rubrics, Lifting MMMU by 2.6 Points

机器之心 · wechat · 2026-09-26

The problem: credit assignment under outcome-only rewards

Multimodal models can produce coherent reasoning yet misread chart numbers or hallucinate objects. If the final answer happens to be right, outcome-based training rewards reasoning that is actually wrong; if the answer is wrong, correct observations inside it are discarded too.

A team from NTU S-Lab, ASTAR and UIUC proposes V-Rubrics, scoring visual answers item by item to fix credit assignment. Accepted to EMNLP 2026 main conference; code and data are open-sourced.

Method: writing visual evidence into the reward

Each visual answer is decomposed into atomic criteria across three dimensions:

Each rubric checks a single atomic item, is short and self-contained, and carries an importance label and weight: Essential/Important/Optional get positive weights of 1-5, while Pitfall is marked -1 or -2. Visual QA uses a VF ≫ RC ≈ IF priority.

Data: 50K questions → 352,938 rubrics

V-Rubrics 50K samples 17 public visual sources covering chart/document QA, diagrams, visual math, counting, educational QA and general visual reasoning. An SFT model answers each candidate 8 times; all-correct items are dropped, the rest are bucketed by accuracy: hard 36.1%, medium 50.4%, simple 13.6%.

Gemini-3-Pro generates 352,938 rubrics over the final 50,248 samples: VF 209,436 (59.3%), RC 101,369 (28.7%), IF 42,133 (11.9%). At training time the verifier reads only the generated response and the self-contained criteria, not the original image.

Training: avoiding reward collapse

Answer correctness and rubric satisfaction each contribute 0.5 to the semantic reward, plus a standard format reward. Answer scores and each positive rubric score are normalized separately within the same rollout group: the final-answer signal covers the whole response, while each rubric keeps an independent advantage. The verifier returns the supporting evidence sentence for each judgment, located by fuzzy matching, and that rubric's advantage applies only to the prefix from the start of the response through that sentence; if no sentence can be located, the advantage falls back to the full response. Triggered Pitfalls veto the answer credit and positive rubric credit for that response.

Results

Starting from Qwen3-VL-8B-Instruct, SFT'd on OpenMMReasoner-SFT-874K, answer-level GRPO and rubric GRPO are trained from the same checkpoint on the same data and sampling budget:

Ablation: adding rubric scoring but still aggregating into a scalar sequence-level reward lifts Overall Avg. from 66.25 to 67.74 (+1.49); component-wise normalization plus prefix localization adds another 0.30 to 68.04.

Examples

On a portrait question, the answer-level model correctly counts 4 people but infers from appearance that one man is over 70 and born before 1945, answering 3; the rubric-trained model recovers the age-to-birth-year reasoning. On a function-graph question, the answer-level model misreads where the function first reaches 2 as x=4, while the rubric model re-reads the graph segment by segment and gets x=2.

Original post →

More from Models

Models channel →