SWEeper-Bench: Best of 15 Frontier Agents Solves Only 59% of Interactive Bug-Finding Tasks
zlab-princeton · hf · 2026-10-15
Princeton's SWEeper-Bench measures whether coding agents can discover interaction-dependent bugs in software they build, not just fix reported ones. It comprises 200 tasks from real bug fixes in 200 popular open-source web apps, where agents get only the codebase and an open-ended request and must design interactive tests; an agentic verifier judges repairs by what a user would observe in the browser. The best of 15 frontier agents passes just 59.0%, with trajectory analysis showing the bottleneck is identifying buggy behavior—agents reach the feature but never trigger or recognize the failure.
More from coding & agent
- CognitionX Emirates 2026 talk: RAG but for tools, mastering the Foundry toolbox — lee_stott · 2026-10-15
- 9 reasons your open-source PR is getting ignored, including 100% AI-generated code — jackedAJ · 2026-10-15
- Claude wrote its own Direct3D 9 and x86 emulators to run kkrieger in a browser — austinc3301 · 2026-10-15
- User hands Google login and SMS 2FA to AI agent, reigniting credential-handoff debate — jonstephens85 · 2026-10-15
- Grok Bot writes its own Apple Desk skill to control Mac apps, Musk amplifies — elonmusk · 2026-10-15
- Thorsten Ball: latest models make context-management tricks like /btw and forking obsolete — andreisavu · 2026-10-15