Meta's SWE-sweep: Best Models Fix Only 4.7% of Hidden Bugs
Meta, together with researchers from Harvard, Stanford and UW, released SWE-sweep, a benchmark that inverts the premise of traditional coding evaluations: instead of giving an agent a ticket describing a known bug, it hands over a real codebase and asks the agent to autonomously find and fix as many hidden bugs as possible. The best model fixed only 4.7%, showing that proactive bug-finding remains far from practical. The code is open-sourced on GitHub.
Confirmed
- The benchmark was jointly released by Meta (Superintelligence Labs / Meta Code) with Harvard, Stanford and UW researchers; Ofir Press and Jack Rae participated in or shared the announcement
- It covers 100 real repositories, 22 languages, and roughly 4.1k (4100+) real defects, including projects like numpy and the PHP interpreter
- Unlike SWE-bench-style benchmarks, no bug location or issue description is provided; models must find bugs unsupervised
- Top models fix under 5% unsupervised, with the best at 4.7%
- Code is open-sourced at facebookresearch/swe-sweep, a lightweight wrapper around Harbor with evaluation tasks and an automatic validation environment for reproducibility
Why it matters
- Existing SWE benchmarks only test fixing known bugs from tickets, while real engineering value lies in finding problems before users hit them; SWE-sweep fills this evaluation gap
- The 4.7% result shows a huge gap between current frontier models and reliable autonomous code review, defining clear headroom for research
2026-10-01 ~ 2026-10-03 · 6 related posts
Primary sources
- [source] SWE-sweep benchmark: top models score under 5% finding bugs unsupervised — OfirPress · 2026-10-01
- [source] SWE-sweep: 4,100 real bugs across 100 repos, best model fixes only 4.7% at $7,230 — OfirPress · 2026-10-02
- [source] SWE-sweep benchmark code open-sourced under facebookresearch on GitHub — OfirPress · 2026-10-02
- Meta's new SWE-sweep eval makes coding agents find bugs in real repos with no hints — jack_w_rae · 2026-10-02
- SWE-sweep benchmark tests whether AI agents can find bugs before users hit them — klieret · 2026-10-02
1 near-duplicate retellings: klieret