CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models
Aneesh Komanduri, Karuna Bhaila, Xintao Wu
EMNLP 2025 Main)
cs.LG, cs.AI, cs.CL
2025-05-21
Tests 9 LVLMs on visual causal reasoning across three physical systems. Causal graphs are easy (100% on the pendulum); propagating an intervention's downstream effects is where they fall to 50-60%.
LVLMs (Large Vision-Language Models) are strong at recognition and visual question answering, but no one had systematically tested their causal reasoning. Text-only work already showed LLMs ace the L1 observational rung of Pearl's causal hierarchy and stumble on L2 intervention and L3 counterfactual. That was measured on text. Whether a model can look at an image and tell which variable moved, which drives which, and what would happen if you changed a value was almost a blank. Existing visual causal benchmarks like CELLO only probe scene-level human-object relations, not formal causal mechanisms. CausalVLBench fills that gap with physically grounded systems whose causal mechanisms are deterministic and checkable.
Three datasets adapted from causal representation learning, each with four causal variables:
Each dataset is split 40% support set (in-context examples) and 60% query, with 1000 sampled per run.
Three tasks map to Pearl's rungs:
Eight open models (LLaVA-OneVision-7B through Qwen2.5-VL-32B) plus Gemini-2.0-Flash. The authors flag upfront that GPT-4V and Claude were skipped due to compute.
On the most physical system, structure inference is almost trivial. Qwen2.5-VL-32B and Gemini both recover a perfect 100% causal graph on the pendulum (SHD=0). Move to the artificial circuit and the best accuracy drops to 73.2% (Gemini), with SHD climbing to around 3.
| Task (zero-shot, Gemini) | Pendulum | Water Flow | Circuit |
| Structure accuracy | 100% | 91.6% | 73.2% |
| Intervention target, 0/8-shot | 39.4 / 47.4 | 37.6 / 55.7 | 10.5 / 66.1 |
| Counterfactual, 0/8-shot | 83.4 / 86.5 | 80.3 / 88.3 | 97.0 / 97.4 |
Intervention target prediction is the real wall. Tasks needing multi-image input sink most models, and adding shots often makes them worse. Gemini climbs the circuit from 10.5% zero-shot to 66.1% at 8-shot, one of the few to benefit from in-context examples; most open models collapse with more context.
Counterfactual numbers look high, but the authors deflate the score: models inflate accuracy by predicting the initial state, not by propagating the intervention. When the intervened variable has descendants (move the light and the shadow should follow), accuracy falls to 50-60%; only leaf-node interventions are answered well.
Chain-of-thought (CoT) prompting splits wide open. Qwen2.5 jumps on circuit structure from 54.8% to 98.5%, but Gemini gets worse with CoT, a long reasoning chain throwing it off.
It draws a sharp line: recognizing correlation and reasoning causally are different skills. Structure inference can be gamed by pattern matching, especially on familiar physics like a pendulum. Intervention and counterfactual demand the model actually understand "if I change X, what follows," and no current model passes.
For anyone building evals or agents, this pins two concrete deficits: multi-image reasoning and counterfactual propagation. If your application needs a model to watch a feed and reason about which valve change causes what, or to reason from medical imaging about how a different treatment would play out, current LVLMs are not there yet.
It is also a methodological warning. An 80%+ counterfactual score can be built on lazily echoing initial values. Eval designers need metrics that separate real propagation from repetition, and the authors' trick of splitting results by whether the intervened node has descendants is exactly that.