Tests Show SWE-bench Multimodal v2 Isn't Actually Multimodal
Blogger dejavucoder demonstrated that SWE-bench Multimodal v2 is not truly multimodal: GPT-6 Luna (xhigh) scores nearly identically with original images, OCR text, or images removed entirely, exposing serious design flaws in the benchmark.
2026-10-21 ~ 2026-10-21 · 3 related posts
- SWE-bench Multimodal v2 isn't multimodal: removing images barely changes model scores — dejavucoder · 2026-10-21
- SWE-bench Multimodal v2 called out: benchmark works fine without the images — dejavucoder · 2026-10-21
- Blogger argues SWE-bench Multimodal v2 is not actually multimodal — dejavucoder · 2026-10-21