The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
Zhiheng Wang, Bo Peng, Lai Wei, Chaochao Lu
cs.AI
2026-08-07
A causal audit of six MLLMs that crop-and-zoom finds most tool calls don't causally affect the answer; gains concentrate in a small calibrated minority of rollouts.
The "thinking-with-images" paradigm lets multimodal LLMs actively crop and zoom into images mid-reasoning, and it's become the default recipe for boosting fine-grained visual understanding. But there's a puzzling pattern: models that call these tools burn several times more tokens than direct inference, yet the accuracy payoff is often marginal, and on some questions tool use actually does worse than just answering directly. Prior work has flagged this "marginal-improvement paradox," but it stops at aggregate accuracy. The harder question is whether each returned crop actually changes what the model answers.
The authors cast visual tool-use as a causal graph over the image, the query, each step's tool action Ti, the returned observation Oi, and the final answer Y. Two path types matter: an observation-mediated path where the crop's visual content genuinely reaches the answer, and an action-induced shortcut where merely emitting a tool call shifts the answer regardless of what's in the crop. They probe this with three interventions of increasing granularity:
Across six models (DeepEyes, Pixel Reasoner, Mini-o3, Qwen3-VL-4B/8B, Thyme) and five perception benchmarks, the policy-level ATE swings wildly: near zero for DeepEyes, moderate (+3 to +7pp) for Pixel Reasoner, Qwen3-VL-4B, and Thyme, and largest for Mini-o3 and Qwen3-VL-8B, reaching +21.3pp and +11.7pp on VisualProbe respectively.
Step-level VEG explains why. DeepEyes' calls sit near zero VEG almost everywhere, even the single best call in a trajectory contributes nothing, meaning its crops are never really used for reasoning. Mini-o3's calls are mostly low-information but occasionally hit something useful. Qwen3-VL-8B's usefulness depends heavily on timing: calls made after confidence is already high (probability gap above 0.95) are mechanically capped near zero gain.
Building on these features, the authors construct a deterministic classifier sorting every rollout into one of four groups: No-call, Mode 1 (Calling Without Looking), Mode 2 (Looking Without Planning), and Calibrated. Decomposing the policy-level ATE on V shows that almost all of the positive gain traces back to the small Calibrated group; the other three groups contribute near zero or cancel out.
Rising aggregate accuracy doesn't mean the tool call itself is doing the work. Looking only at aggregate numbers, it's easy to conclude a thinking-with-images model has learned to use tools well when it's actually just learned that calling a tool feels safer, without learning when to look or when to stop. The authors' working hypothesis is that outcome-only reinforcement learning naturally reinforces "call and get rewarded" shortcuts while never penalizing calls that carry no information or that fail to stop. The diagnostic they build is directly actionable: skip Mode 1 trajectories at inference time, add early-stopping rules for Mode 2, or use VEG as a process-level reward signal.
The three intervention levels have different access requirements: policy- and trajectory-level tests only need to run the model and control what it sees, so they work on black-box systems, but step-level VEG needs token-level probabilities and thus white-box access, so the paper can't confirm whether closed models like OpenAI o3/o4-mini show the same pattern. The study covers only crop-and-zoom; whether OCR, video frame selection, or other tools show similar miscalibration is untested. The RL-trap explanation is a hypothesis consistent with the observed data, not something the paper establishes through controlled training runs that vary only the reward signal; the authors leave that for future work.