ValsAI launches SWE-bench Multimodal leaderboard focused on visual software bugs
OfirPress · x · 2026-10-15
ValsAI launched a leaderboard for SWE-bench Multimodal, a challenging SWE-bench variant focused on visual bugs (e.g. a map not rendering correctly or a button on the wrong side). Per author OfirPress, even on this relatively simpler bench there's still lots to do. The project, led by jyangballin and closji, is now fully open source with 480 tasks available for local evaluation (Multimodal v2).
Related event: SWE-bench Multimodal Leaderboard Debuts, Claude Opus 5.5 Leads at 70.21%(4 posts)→
More from Models
- Nathan Lambert: distillation buys a few months, Chinese labs lack compute not talent — natolambert · 2026-10-15
- Argon hits cyber, CBRN CCLs and misalignment TCL; report flags CoT transparency — vkrakovna · 2026-10-15
- First Proof batch 3: mathematicians grade GPT-6, Opus 5.5 and Kimi K3 on research-level proofs — littmath · 2026-10-15
- Google tops Signal65 PINNACLE agentic leaderboard: Gemini 4 Argon at $0.87 per correct task beats Sonnet and GPT — ryanshrout · 2026-10-15
- DeepSeek V4.1 Flash tops cost-efficiency at $0.134 per task, leaderboard shows — bindureddy · 2026-10-15
- Gemini's New Argon Image Model Keeps Generating Distorted 'Little People' — RecognitionHefty · 2026-10-15