See2Think: Do Multimodal Models Really Use Intermediate Visual States?
Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang
cs.CV, cs.AI
2026-07-29
Do multimodal models really use the images they draw while reasoning? They pick relevant actions but render them poorly; corrupting a returned image drops 3D accuracy 15.5 points.
Multimodal models have picked up the habit of "thinking with images": while solving a problem they draw auxiliary lines, crop or highlight regions, or call tools to produce intermediate images, then reason on from there. The open question is whether those intermediate visual states actually carry the reasoning, or are just decoration. Prior visual-reasoning benchmarks grade only the final answer, or at best judge how plausible the visual trace looks. None can tell you whether the model's next reasoning step genuinely used the picture it just drew. See2Think turns that "genuine use versus decoration" question into something measurable.
The framework has two parts. See2ThinkBench is the data side: 1,200 problems that cannot be solved without looking at the image, across 12 categories spanning 2D structured reasoning (geometry, spatial puzzles, physics, chemistry), 3D scenes, and real-world tasks (robot manipulation, intuitive physics). Construction filters out text-shortcut cases: a strong model captions each image, a second model gets only the question plus that caption (no image) and is queried five times, and any sample it answers correctly three or more times is thrown out.
The core is the VAoT (Visual Action-of-Thought) protocol. It records reasoning as an alternating loop of textual thought, visual action, rendered image, and further reasoning, under four controlled settings:
Three process scores isolate where things break: Action Relevance (does the action target the right evidence), Render Faithfulness (does the renderer carry out the action as asked), and Feedback Uptake (does later reasoning actually use the returned image). The corruption setting then separates two ideas that are usually conflated: whether a visual state is useful (correct rendering raises the score) and whether the model depends on it (corruption lowers the score).
Four models (GPT-5.5, GPT-o3, Gemini 3.5 Flash, Qwen3-VL-32B-Instruct) ran all four settings on all 1,200 problems. The headline is counterintuitive: no setting dominates. GPT-5.5 and Qwen3-VL-32B do best with plain CoT; GPT-o3 peaks under VAoT (real rendering); Gemini 3.5 Flash peaks under VAoT-NoRender (planning without rendering). Whether visual thinking helps depends on the model and the task.
The process diagnosis explains why. Action Relevance is near saturation for every model (0.96 to 0.99 overall); the models generally know where to look. Render Faithfulness sits around 0.6. The bottleneck is not choosing the action but executing it faithfully and then using the result. In 3D scenes the correct-versus-incorrect gap is largest on Render Faithfulness (0.097), dwarfing the Action Relevance gap (0.011).
| Dimension (overall) | GPT-5.5 | GPT-o3 | Gemini 3.5 Flash | Qwen3-VL-32B |
| Action Relevance | 0.985 | 0.978 | 0.976 | 0.958 |
| Render Faithfulness | 0.616 | 0.614 | 0.630 | 0.594 |
| Feedback Uptake | 0.772 | 0.830 | 0.694 | 0.871 |
The sharpest finding comes from the corruption experiment. The higher a sample's Feedback Uptake, the more its accuracy collapses when the returned image is corrupted. In 3D scenes, as Feedback Uptake rises from 0 to 1, the accuracy drop climbs from 3.5 to 15.5 points. Across the four models, 32.7% to 54.2% of samples change their final answer purely because the picture was swapped for a corrupted one. Yet correct rendering (VAoT) brings no matching net gain over not rendering, and is often lower. The models do depend on these visual states, but the dependence helps little, and often they lean on images that are wrong.
The value here is not which model scores highest. It is the diagnostic method. If you build multimodal agents that call visual tools, watching only the final accuracy is misleading; a model can look like it is using images while doing nothing of the kind. Two distinctions from See2Think are practical: the bottleneck lands on rendering and feedback use rather than action selection, and "useful" and "depended-on" are different axes. For anyone working on visual tools, image-editing renderers, or process rewards, Render Faithfulness is a hard metric worth optimizing on its own.
The authors list three: only four models were tested; process scoring depends on an external renderer and an automatic judge (GPT-5.4), which a 240-trajectory human audit found 92.9% to 96.9% reasonable-or-partial but is still fallible; and the WrongRender effect size depends on corruption quality, with only 56.7% of interventions passing strict audit, so the 15.5-point figure is an order of magnitude rather than a precise number.
One gap stands out beyond the authors' list. Across all models the winning setting is always CoT or NoRender, the ones that skip real rendering; VAoT almost never wins. That raises a question the paper leaves open: is the external renderer simply not good enough? A more reliable execution backend might let real rendering pay off. The study does not isolate that variable.