SWE-bench Multimodal v2.0 Launches with 480 Open-Source Visual Coding Tasks

Stanford and Meta MSL jointly released SWE-bench Multimodal v2.0, an open-source benchmark of 480 visual coding tasks available through the SWE-bench repository with an open leaderboard. Coding agents must interpret screenshots, charts and audio recordings alongside code to diagnose and fix repository bugs.

Confirmed

v2.0 contains 480 visual coding tasks, available via the SWE-bench repository with open leaderboard submissions
v1 was closed-source, limiting adoption; v2 is open-source with cleaned data, removing broken tasks and flaky tests
The project was a cross-team effort between Stanford and Meta MSL
@OfirPress noted that unlike Verified and Multilingual variants, this version emphasizes frontend development tasks, with most requiring visual capabilities

Why it matters

@OfirPress considers it the most challenging SWE-bench variant, adding vision to coding evaluation
@jyangballin observed steady multimodal gains from Anthropic models, suggesting significant room for improvement
The open-source release should boost community adoption and enable more reliable comparison of multimodal coding agents

2026-09-01 ~ 2026-09-01 · 5 related posts

Primary sources

SWE-bench Multimodal v2 Released: 480 Visual Coding Tasks, Open-Sourced — jyangballin · 2026-09-01 22:54 (UTC+08:00)
SWE-bench Multimodal Released by Stanford and Meta, Leaderboard Open — jyangballin · 2026-09-01 22:54 (UTC+08:00)
SWE-bench Multimodal variant focuses on front-end development with vision tasks — OfirPress · 2026-09-01 23:16 (UTC+08:00)

SWE-bench Multimodal v2 Released: 480 Visual Coding Tasks — jyangballin · 2026-09-01
[source] SWE-bench Multimodal Released by Stanford and Meta, Leaderboard Open — jyangballin · 2026-09-01
[source] SWE-bench Multimodal variant focuses on front-end development with vision tasks — OfirPress · 2026-09-01

2 near-duplicate retellings: jyangballin · xeophon