Scale can't fix VLM reasoning: EMNLP paper shows reporting bias is the bottleneck
RanjayKrishna · x · 2026-10-27
The TACL/EMNLP 2026 paper "Scale Can't Overcome Pragmatics" argues that VLM reasoning limits stem from reporting bias: humans systematically omit spatial, counting, negation, and temporal language when captioning images—e.g. "blue" appears in only 2% of LAION-2B captions. A new benchmark testing these concepts shows most VLMs near random chance, and unlike perception, scaling does not improve reasoning. Mitigation requires intentionally instructing annotators to include omitted behaviors. Conclusion: scale suffices for emergence of recognition, but not reasoning.
More from Models
- GPT-6.1 Terra rolls out to all Plus, Pro, Enterprise and Edu users — OpenAI · 2026-10-27
- OpenAI launches GPT-6.1 Terra with 50% lower API prices than GPT-5.6 Terra — OpenAI · 2026-10-27
- GPT-6.1 Terra API pricing revealed: $1 input / $5 output, $0.05 cached — airesearch12 · 2026-10-27
- In cyber, intention matters little: Chintala on model info leaks, and what users should do — brhydon · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27