Meta's SWE-sweep: Best Models Fix Only 4.7% of Hidden Bugs

Meta, together with researchers from Harvard, Stanford and UW, released SWE-sweep, a benchmark that inverts the premise of traditional coding evaluations: instead of giving an agent a ticket describing a known bug, it hands over a real codebase and asks the agent to autonomously find and fix as many hidden bugs as possible. The best model fixed only 4.7%, showing that proactive bug-finding remains far from practical. The code is open-sourced on GitHub.

Confirmed

Why it matters

2026-10-01 ~ 2026-10-03 · 6 related posts

Primary sources

1 near-duplicate retellings: klieret