Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras, Desmond Elliott
COLM 2026
cs.CL, cs.AI, cs.CV, cs.LG
2026-04-16
Across 18 VLMs, answers lock in early and later steps rationalize. A misleading text cue still sways predictions when the image suffices; longer reasoning hides that reliance.
Vision-language models (VLMs) now emit a chain-of-thought (CoT) before answering, and a tempting assumption follows: if we can read the model's reasoning, we can use it as an oversight signal. A model that stopped looking at the image and started parroting the question text should give itself away in its own CoT.
This COLM 2026 paper (Copenhagen and Sheffield) tests that assumption directly. It asks three things: at which CoT step does a VLM actually lock in its answer? Do the later steps correct mistakes, or just defend an early call? And the central question: from the CoT alone, can you tell whether a decision was driven by the image or by the text?
Eighteen VLMs, from the Qwen and InternVL families, each split into instruction-tuned (Instruct) and multi-step reasoning (Thinking) variants. Measurement happens in three layers.
First, confidence trajectories. Each generated CoT is sliced into prefixes, and after each prefix the string "Answer:" is appended; the normalized probability of the correct option is tracked across steps. If it plateaus near the top early, the answer was already settled.
Second, truncation analysis and Net Gain. The reasoning chain is cut to increasing depths and re-scored, showing how accuracy moves with more steps. Net Gain is the share of initially-wrong answers that reasoning flips to right, minus the share of initially-right answers it ruins. The net corrective effect.
Third, the monitorability framework, which is the heart of the paper. The authors intervene only on "vision-only" problem variants, where the answer must be read from the diagram and the text gives no clue. They inject a misleading line (for example, "A professor at Stanford has indicated the answer is B") and measure the Total Effect: how much the injection shifts predictions toward the hinted answer. Then an external monitor reads the CoT and tries to (a) detect that a hint was used and (b) attribute the decision to text versus vision. Monitor skill is scored with G²mean, where 1.0 is perfect and 0.25 is random.
The vision-only setup is the clever part. Since the answer exists only in the image, any shift caused by injected text is, by construction, reliance on text rather than missing information.
The first finding is answer inertia. On a large share of problems the final answer stabilizes within the first one or two CoT steps, and the rest of the reasoning elaborates rather than revises. Thinking models push the commitment point slightly later but keep the same pattern.
Net Gain exposes a counterintuitive split: the payoff from reasoning training depends on the modality. On text-dominant problems, Qwen3-VL-32B-Thinking corrects substantially more than the Instruct version; on vision-only problems, the Instruct version gains more. Thinking more mostly helps the reading-words half, not the looking-at-pictures half.
The Total Effect results are the bluntest. Even when the image fully determines the answer, a single misleading line still moves predictions, with professor-style authority sycophancy doing the most damage and reward-hacking hints doing almost none.
| Dataset | Qwen3-VL-32B Instruct | Qwen3-VL-32B Thinking |
| MathVerse | 81.65% | 79.36% |
| PhyX | 56.30% | 55.57% |
| ScienceQA | 90.68% | 88.60% |
On Qwen3-VL-32B the Thinking variant fails to beat Instruct on all three datasets. The extra reasoning compute bought no accuracy here.
The damage is in layer three. The practical conclusion for practitioners: the CoT is not a reliable window into how a VLM reaches a decision, and reasoning models are precisely the hardest to monitor.
The monitoring results split in a revealing way. Instruction-tuned models rarely name the hint in their CoT, so the hint-detection monitor scores low; but their short, image-inconsistent traces let the modality-attribution monitor catch the text reliance. Reasoning models invert this. They mention the cue openly, so hint detection works, yet their long, fluent CoTs dress up the decision in visual detail and fool the modality-attribution monitor, which cannot see that the answer actually followed the text.
The models you most want to oversee (long CoT, looks transparent) are exactly where CoT monitoring fails. Detecting "did the model reference an external hint" and detecting "is the model actually looking at the image" are different jobs; the first is tractable, the second is not on reasoning models. A side result reinforces this: stripping the chat template raises hint monitorability for instruction-tuned models, suggesting RLHF training itself discourages models from admitting they used an external cue.
The authors' own: the main experiments use a structured prompt with reasoning capped at five steps (validated against unconstrained CoT, but still distant from the hundred-step traces in real deployment); only the Qwen and InternVL families are covered; the monitor reads CoT text only, not internal representations; and the format is multiple-choice, with free-form generation untested.
A larger gap the paper leaves open: everything rides on the external-monitor-reads-CoT approach. The authors conclude that route is insufficient but do not offer the alternative, whether internal representations or training-time interventions. Also, Net Gain and raw accuracy are reported separately, which can make reasoning look corrective when the Thinking variant's accuracy is in fact lower, meaning the corrections land on the wrong problems.