SWE-bench Multimodal v2 called out: benchmark works fine without the images
dejavucoder · x · 2026-10-21
Blogger dejavucoder published a post titled "SWE-bench Multimodal v2 is not multimodal", arguing the benchmark has a fundamental design flaw.
The validation heuristic is instructive: a good test for multimodal evals is running them without the images — if scores barely change, the visual inputs aren't actually needed. That's exactly what happened with SWE-bench Multimodal v2, showing it doesn't genuinely require multimodal input.
xeophon and others in the evals community amplified the finding, urging anyone interested in LLM evals and benchmarks to take note.
Related event: Developer Claims SWE-bench Multimodal v2 Is Not Actually Multimodal(5 posts)→
More from Research
- Xaira Unveils X-Scientist: AI Nominates Drug Targets ~10x Faster Than Humans — BoWang87 · 2026-10-22
- CT-Translate predicts immunotherapy response: top-ranked third hits 61% response across 451 patients — david_van_dijk · 2026-10-22
- SuperMimic trains humanoid control on 129k motions with 1 GPU in 3 hours, cutting GPU-hours 1000x — kevin_zakka · 2026-10-22
- RUBATO paper: decoupled clocks let full-duplex models think at full speed — EliasEskin · 2026-10-22
- OpenAI models' real-world AI R&D time horizon is only 1/12 of METR's benchmark suite — tobyordoxford · 2026-10-22
- ICLR 2027 submissions flooded with AI slop? CS professor pushes back on the narrative — CSProfKGD · 2026-10-22