SWE-sweep: Meta's new benchmark shows top models fix under 5% of 4.1k real bugs
eliebakouch · x · 2026-10-19
Meta Superintelligence Labs (with Harvard, UW, Stanford) launched SWE-sweep, a benchmark where agents must find and fix bugs across 100 real repos (4,100 bugs) with no hints. Leading results with mini-SWE-agent: OpenAI Sol 5.6 (xhigh) resolves 4.7% for a steep $7,230; Luna 5.6 (xhigh) hits 2.5% for just $224; Anthropic Opus 5 (xhigh) manages 1.3% at $5,363; Kimi K3 0.6%; Gemini 3.5 Flash Lite 0.1%. The near-flat scores highlight how hard open-ended debugging remains. The post also amplified an unconfirmed report that Anthropic is nearing a "Fable 5.5" release, claimed to infer users' underlying intent and generalize single-bug fixes to whole bug classes.
More from Models
- OpenAI upgrades open-source devs to $200/mo ChatGPT Pro, free for six months — NielsRogge · 2026-10-20
- Musk Endorses Grok Bot: 'It Might Turn Out to Be Your Favorite Employee' — elonmusk · 2026-10-20
- Gemini Argon Users Slam Usage Limits: One Slide Deck and You're Capped — zizo999 · 2026-10-20
- Spend every token in your quotas now — generous AI allowances won't last, dev warns — gethackteam · 2026-10-20
- DeepSeek V4.1 Flash 1017 hits 45 on AA Intelligence Index, tying #2 open-weights — ArtificialAnlys · 2026-10-20
- Debate: if Chinese labs are so efficient, why do they always lag the frontier? — kuza55 · 2026-10-20