ShadowBench: 19 VLMs fail at judging shadow physical consistency, only Gemini-3.1-pro beats chance
HirokatuKataoka · x · 2026-10-28
A WACV 2027 paper introduces ShadowBench, built from real photographs with correct-shadow, shadow-removed, and human-annotated wrong-direction states, probing VLMs on three shadow reasoning tasks:
- Reading visible shadow direction: well above chance and improves with scale.
- Inferring absent shadow direction: weak, only 46-51%.
- Judging physical consistency: all 19 open VLMs (2B-78B) and frontier proprietary models fall to chance under a bias-free two-alternative test; only Gemini-3.1-pro modestly exceeds chance, far below humans. Chain-of-Thought doesn't help.
The paper shows that borrowing the missing shadow prior from image generation models lifts binary validity accuracy from 50% to 74.7% and helps expose physically inconsistent images. The missing ingredient is an implicit physical prior linking illumination, geometry, and cast shadows.
More from Models
- Researcher Drops Claude 5.5 for Biology Work Over Too Many Refusals — anshulkundaje · 2026-10-28
- Claude Opus 5.5 beats Opus 5 in 14 of 16 comparisons, leads physical agent Elo 1313-1289 — qinzytech · 2026-10-28
- Stanford prof: LLM reviewers catch mistakes scientists keep missing — anshulkundaje · 2026-10-28
- Correction: DeepSeek-V4.1-Flash-1017 scores only ~28% on FrontierCode 2.0 — scaling01 · 2026-10-28
- LiveBench update: Claude Fable 5.1 tops real-world leaderboard at 85.7, DeepSeek leads open source — bindureddy · 2026-10-28
- Open-source coding models look benchmaxxed; dev hopes FrontierCode 2.0 stays clean — zyxciss · 2026-10-28