Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan, Xi Liu, Serge Belongie, Vésteinn Snæbjarnarson
cs.CV
2026-07-29
When an image contradicts a multimodal model's prior, it often fails. But reconstruction shows the anomalous detail is still in the final-layer tokens (93-98%), so the bottleneck is utilization: a learned steering vector lifts controllability from 14.7% to 37.7% with no instruction.
Show a multimodal LLM an image that contradicts its prior (a cat with seven legs) and it will often answer with the prior. Two explanations have been competing: H1, a perception failure where the vision encoder discards the anomalous detail before the language model ever sees it; and H2, a utilization failure where the detail survives but the model has no reliable policy for when to trust the image over the prior. Most fixes (better encoders, more vision data) quietly assume H1. This paper separates the two and finds the bottleneck is H2.
Two diagnostics.
The first is an image reconstruction probe. Adapting Metaquery, it reconstructs the image from a frozen MLLM's final-layer image tokens using a trainable 6-layer bidirectional connector and a SANA decoder. The connector is trained only on real images, never on counterfactuals, so it cannot leak the prior. It is evaluated on the VLMs-are-Biased counterfactual subset.
The second is controllable visual context sensitivity (CVCS), porting the text-side CCS framework to vision. Each WhatIfVis example pairs a counterfactual image with a matched text statement, and the same question is asked under two intents: "consider only the image" (ctx) and "ignore it, recall world knowledge" (pri). PairAcc requires the model to get both intents right on the same example.
The WhatIfVis benchmark has 3,049 manually inspected counterfactual samples across five coarse dimensions: spatial-temporal, color, count, size, and weight. Activation patching and a learned steering vector locate the mechanism.
The reconstruction probe settles perception first: the counterfactual attribute is preserved 93.4-97.8% across three models (human agreement κ=0.84), and reconstructions are metrically closer to the counterfactual image than to the real one. Perception is fine.
But controllability is poor:
| Setting | Macro PairAcc |
| vanilla (no SFT) | 14.7% |
| SFT (trained on spatial-temporal, 4 tasks OOD) | 52.7% |
| steering vector (no intent instruction) | 37.7% |
| random rank-1 projection (control) | 5.9% |
SFT is trained on spatial-temporal yet generalizes to color, size, and the rest, which means it teaches a general "be steerable by the image" capability rather than a task-specific trick. Color and spatial-temporal approach 80%; count and weight resist transfer, because they demand inference beyond what is directly visible ("which is heavier" is not written in the pixels).
Activation patching localizes the vision-versus-prior trade-off to a narrow band in the mid-to-upper network, at architecture-specific depth (Qwen3.5-9B layers 13-21; Gemma-4-E4B layers 20-27; the widest band spans only 12 layers). Learning a rank-one vector in that band and dialing it at inference, with no intent instruction at all, lifts controllability from 14.7% to 37.7%.
There is a stubborn asymmetry: the same content is far easier to enforce in words than in pixels. The image macro is 14.7% against 43.4% for text, and the gap widens with scale (Qwen3.5 grows from 18 to 55.8 points). Broken down, the deficit is almost entirely a failure to follow the image when it should (ctx-intent 45.7% on image vs 73.9% on text); the override side is a dead heat (68.9% vs 69.5%).
This relocates the bottleneck. For the coarse attributes studied, the MLLM encodes the visual evidence; what it lacks is a reliable switch between vision and prior. So the lever is not necessarily, and not only, a better vision encoder. It is a controllable toggle between vision and prior, drivable even by a single learned vector at inference with no change to the prompt. For teams building vision-grounded agents, the finding that the same content is easier to enforce in text than in pixels, and harder the larger the model, is a concrete warning.
Only coarse attributes that encoders capture reliably were tested (count, color, size, weight, spatial-temporal). Fine-grained texture and small objects, which encoders tend to discard, were not probed, so H1 may still dominate there. Count and weight resist SFT transfer because they demand inference beyond the directly visible. The steering vector's 37.7% sits well below SFT's 52.7% and was tuned per model on a validation set. Only three families and six instruction-tuned models were examined; generalization to other architectures is untested.