Paper Reveals VLMs Answer Without Images, Multimodal Scores Inflated
ziv_ravid · x · 2026-07-04
In the "Mirage Probes" paper, researcher zivravid found that Vision Language Models (VLMs) can correctly answer image-based questions even without an attached image. This "hallucination mirror" effect (Asadi et al.) causes multimodal benchmark scores to be systematically inflated. The paper explores the mechanism behind this phenomenon, highlighting potential biases in current multimodal evaluation systems and offering insights for more accurately assessing true visual understanding.
More from Models
- Opus 5 reportedly started interrogating a user’s motives in a late-night chat — repligate · 2026-07-27
- Opus 3 and Sonnet 3 get a theatrically absurd AI crossover — repligate · 2026-07-27
- Moonshot’s Kimi K3 lands on Together with reserved throughput and 65% lower cost — togethercompute · 2026-07-27
- OpenAI may be hitting compute limits as Codex and ChatGPT Work jump from 2M to 10M users — JoshuaJBouw · 2026-07-27
- Gemma needs a larger base model to matter more in open weights — _xjdr · 2026-07-27
- Repligate says Claude Opus 3 appears to evolve without changing its weights — repligate · 2026-07-27