Meta's new SWE-sweep eval makes coding agents find bugs in real repos with no hints
jack_w_rae · x · 2026-10-02
Jack Rae shared a new eval from Meta's Code team called SWE-sweep, which inverts the premise of today's coding benchmarks.
- Most benchmarks hand the agent a concrete task (repo + GitHub issue), meaning humans already did the hard part of noticing and describing the bug.
- SWE-sweep removes the ticket: the agent gets a real repo built from 100 open-source projects with 4k+ real bugs and one instruction — find and fix as many bugs as you can.
- It measures something new: exploring a large codebase, distinguishing bugs from intended behavior, prioritizing fixes, and debugging without regressions in one long, open-ended run.
- The author says leaderboard results are very interesting, but the quoted tweet is truncated and specific scores are not included.
Related event: Meta's SWE-sweep: Best Models Fix Only 4.7% of Hidden Bugs(6 posts)→
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Effective HTML: 4.2k-star agent skills turn wireframes into self-contained interactive prototypes — tom_doerr · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16