Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometry
Jiani Guo, Junjie Wang, Jie Wu, Pengxiang Zhao, Dongdong Zhang, Shaohan Huang, Yujiu Yang, Furu Wei
cs.LG, cs.CL
2026-08-31
Tsinghua and MSR train VLMs on line-addressable diagram code; CE-GRPO averages 76.04 on nine geometry benches, +8.09 vs Qwen3-VL-8B and +3.43 vs trajectory GRPO.
Geometry is a harsh multimodal test. The model has to read diagram relations accurately and keep using them across many deduction steps. One bad object binding, one misread angle, one wrong auxiliary line, and the rest of the trace is waste.
Two research lines have stayed apart. Program-aided and formal geometry systems expose structure but still optimize the whole response. Fine-grained RL methods such as VinePPO, GPO, and GRPO-MA localize credit, yet their cut points come from learned values, fixed segments, or token statistics. Missing is a semantic unit that reasoning and learning share.
The authors call that unit credit-addressable reasoning: the decisions shown at inference are the addresses where training compares alternatives and assigns credit.
A six-model study on MathVerse asks whether code can be that unit. Diagram plus externally generated code beats either channel alone for every model. For open models, self-written code trails external code by 11.7 to 16.6 points in the code-only setting. Code should sit beside the image, and the policy has to learn to write it.
Code-CoT installs that protocol. The model keeps the original diagram, emits line-numbered executable Matplotlib perception code, then interleaves four event types: think, reference (cite code lines for a visual fact), auxiliary (extend the figure), and coordinate (set a frame). Those tags are checkers at inference and branch points at train time.
SFT on 18,302 filtered traces teaches the format: Gemini-3.1-Pro transcribes the diagram, DeepSeek-V4-Pro writes the events. The vision encoder is frozen. Prompting is not enough. A Code-CoT prompt drops Qwen3-VL-8B from 67.95 to 49.26.
CE-GRPO (Critical-Event Group Relative Policy Optimization) localizes credit at event boundaries. A structural prior plus type-normalized entropy picks candidates. The image, question, and full prefix before the event stay fixed; several continuations run to the final answer; group-relative differences in the terminal reward update only the regenerated event and what follows. The prefix is prompt, not loss. If every continuation scores the same, the group is a no-op. Ordinary and shared-prefix prompts mix 1:1 over 11,450 RL problems. The reward is programmatic: −1 for invalid structure, otherwise correctness plus 0.3 times action-validity minus repetition and answer-leak penalties, clipped to [−1, 1.3]. Inference is greedy, single pass, no external solver.
On nine geometry benchmarks, CE-GRPO averages 76.04. That is +8.09 over Qwen3-VL-8B, +6.49 over Code-CoT SFT, and +3.43 over trajectory-level GRPO, with gains on all nine. The extra lift sits on long-dependency tasks: +15.16 on GeoLaux-mini and +9.44 on MM-Math versus trajectory GRPO. Visual grounding and standard plane geometry move less, and not always in the same direction.
On validly closed responses the mean gap is still +3.91, so this is solution quality, not format compliance alone. Diagram-to-code fidelity on 100 MathVerse-TD items: macro recall 55.21% (prompt) to 70.16% (SFT) to 80.43% (CE-GRPO); render success 89.0% to 99.0%. On MathVerse's text-dominant to vision-only ladder, the TD–VO gap shrinks from 30.07 to 14.09, with +15.10 on vision-only.
Selector ablation: random 72.48, entropy-only 72.83, structure-only 74.26, structure plus entropy 76.04. Invalid-termination falls from 12.31% to 4.73%. Structural selection finds outcome-changing events about 30% more often than chance. The CE-GRPO margin over trajectory GRPO grows 3.77 points per extra intermediate event (r=0.866, p=0.0016). Trajectory GRPO's own gain over SFT falls 5.55 points per event.
Against the two-stage GDP-4B-RL then Qwen3-VL-8B pipeline, a single call still wins 3.07 on average (seven of nine). Geometry3K trails by 4.42 and PGPS9K by 7.30, where a frozen symbolic parse still helps.
| Method | Nine-benchmark mean |
| Qwen3-VL-8B | 67.95 |
| Code-CoT prompt | 49.26 |
| Code-CoT SFT | 69.55 |
| Trajectory GRPO | 72.61 |
| CE-GRPO | 76.04 |
For long-chain RL, the move is to make the reasoning representation the credit address. Event boundaries come from protocol tags. No process labels, no extra value model. Geometry is a clean test: a bad auxiliary line shows up several steps later.
The deployable object is narrow. Inference is one greedy decode of an 8B VLM. The protocol is heavy; prompting it without SFT hurts. The natural transfer target is tasks with explicit intermediate structure and delayed outcomes, not open-ended VQA by default.
There is no standalone limitations section. Prefixes are harvested once from the SFT policy and frozen, so later RL may branch from a stale distribution. Trajectory GRPO used 8 rollouts per prompt; CE-GRPO used 4, so the compute budget is not matched. Answer grading uses Gemini-3.1-Pro and Gemini-2.5-Pro as judges. SFT traces themselves come from a stronger teacher pair.
Two-stage symbolic systems still lead on Geometry3K and PGPS9K. The claim that the framework extends to other structured reasoning tasks is not measured here.