SWE-sweep benchmark: top models score under 5% finding bugs unsupervised
OfirPress · x · 2026-10-01
- Ofir Press's team released SWE-sweep, a benchmark testing whether LMs can discover and fix bugs without being told what went wrong—key for agents that proactively maintain repos.
- It spans 100 real repos, 22 languages, and 4k real bugs (numpy, PHP interpreter, Lean kernel).
- Top models score <5%, showing labs haven't started climbing this ability curve.
- The authors see it as a north-star benchmark, among their most challenging ever.
Related event: Meta's SWE-sweep: Best Models Fix Only 4.7% of Hidden Bugs(6 posts)→
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16
- AI agent deletes 43,599 lines of OpenClaw test code in one PR — vincent_koc · 2026-11-16