SWE-bench Multimodal v2 isn't multimodal: removing images barely changes model scores
dejavucoder · x · 2026-10-21
While establishing baselines at the Prime Intellect residency, the author noticed GPT-6 Luna (xhigh) scored the same on SWE-bench Multimodal v2 with normal images, OCR output, or no images at all — and wrote up a blog post showing the 'multimodal' benchmark barely depends on images.
Key findings:
- Removing images barely changes scores: GPT-6 Luna xhigh on all 480 tasks scored 36.9% with images vs 37.1% without (+0.2); Sonnet 5.5 max on a 60-task subset went from 35.0% to 40.0% (+5.0, actually higher)
- Most tasks are solvable from the issue text and code alone, since images mostly repeat what the text says
- Hidden tests rarely check image-only details like colors or spacing in design specs
- As a control, MMMU-Pro accuracy dropped 42.8 points without images (79.7%→36.9%), confirming the models' vision works — the benchmark design is the problem
More from Models
- DeepSeek V4.1 Flash, FLUX Klein and more open models land on Hugging Face Inference Endpoints — SergioPaniego · 2026-10-21
- inclusionAI releases Ling-3.0 UBTree speculative-decoding draft models on Hugging Face — jacek2023 · 2026-10-21
- Ornith-1.5-122B open-weights model lands with 84.5% SWE-bench, runs on a single DGX card — nikola_mr64990 · 2026-10-21
- OpenAI says its models found critical PHP flaws powering a fix for ~70% of the web — Kyrannio · 2026-10-21
- Machine Witness: Weekly AI News Reacted to by Seven AI Models — kavanutz · 2026-10-21
- Gemini chat memory has three blind spots; a paste-in prompt fixes precise recall — Ok_Negotiation_2587 · 2026-10-21