Research finds multimodal models rarely actually look at images
MoonL88537 · x · 2026-10-22
A developer shares a striking find: a frontier model (likely Opus) once said very odd things, and it turned out nobody had actually checked the images. Follow-up research uncovered a solid paper concluding that multimodal models across the board rarely actually look at input images, with more recent models being somewhat less prone to this. The experiments originated from ideas by florian and snimu on a purpose-built benchmark.
More from Models
- Anthropic models most complete, OpenAI models more accurate on medical benchmarks — ArtificialAnlys · 2026-10-22
- Anthropic most complete, OpenAI most accurate: 14 models compared on medical benchmarks — ArtificialAnlys · 2026-10-22
- BitKindle: AI bootstraps a full C compiler from a 3,584-byte machine-code seed — dotey · 2026-10-22
- A tokenization-shared open model ladder is sorely missing, and all roads lead to Qwen — bclavie · 2026-10-22
- Chonk DOOM: a DOOM-like shooter built for Mistral Large 4 to beat on its own — GuillaumeLample · 2026-10-22
- DeepSeek V4.1 Pro rumored to train on 50T+ pretraining tokens, matching Microsoft's scale — teortaxesTex · 2026-10-22