Paper Reveals VLMs Answer Without Images, Multimodal Scores Inflated
ziv_ravid · x · 2026-07-04
In the "Mirage Probes" paper, researcher zivravid found that Vision Language Models (VLMs) can correctly answer image-based questions even without an attached image. This "hallucination mirror" effect (Asadi et al.) causes multimodal benchmark scores to be systematically inflated. The paper explores the mechanism behind this phenomenon, highlighting potential biases in current multimodal evaluation systems and offering insights for more accurately assessing true visual understanding.
More from Models
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- giffmana: the env being used in training is part of the point — giffmana · 2026-09-11
- awesome-llm-leaderboards: an open-source directory of LLM leaderboards, pricing tables, comparison tools — Last_Establishment_1 · 2026-09-11
- Anthropic claims it works to keep eval environments unidentifiable to models — MaxKannen · 2026-09-11
- Nex N2.5 Pro released on Hugging Face with 407GB of weights — jinnyjuice · 2026-09-11
- RoMa v2 image matching model unveiled in the usual black poster — ducha_aiki · 2026-09-11