SWE-sweep: 4,100 real bugs across 100 repos, best model fixes only 4.7% at $7,230
OfirPress · x · 2026-10-02
Researchers from Meta Superintelligence Labs, Harvard, UW and Stanford launched SWE-sweep, a benchmark where agents must autonomously discover and fix as many bugs as they can in real repositories — with no hints about bug type or location. It covers 100 repositories and 4,100 bugs.
Highlights from the first leaderboard (all running mini-SWE-agent):
- Sol 5.6 (xhigh) tops the board at 4.7% bugs resolved, but costs $7,230 total ($72.30 per repo, 233 turns)
- Luna 5.6 (xhigh) resolves 2.5% for just $224 — the best cost-efficiency
- Opus 5 (xhigh) resolves 1.3% at $5,363; Kimi K3 resolves 0.6% at $2,451
The code is fully open source, and the team says automatic checks guard against fixes introducing regressions. The benchmark shows frontier models are still very weak at autonomously hunting bugs in large real-world codebases.
Related event: Meta's SWE-sweep: Best Models Fix Only 4.7% of Hidden Bugs(6 posts)→
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Effective HTML: 4.2k-star agent skills turn wireframes into self-contained interactive prototypes — tom_doerr · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16