Paper: Coding Agents Find Under 20% of Bugs Humans Fix, but Report Many Valid New Ones
OfirPress · x · 2026-10-29
A new paper studies proactive bug-finding agents against a holdout set of human-fixed bugs: agents recall fewer than 20%, yet most of the many additional bugs they report are valid — a clear distributional mismatch with maintainers. Cost data: DeepSeek V4.1 Flash hits F1 10.5 for $73, GPT-5.6 Sol (high) tops recall at 20.8% but costs $1,185, and GPT-5.6 Luna (low) runs just $8.
Related event: Coding Agents Find Different Bugs Than Maintainers Fix, Paper Shows(3 posts)→
More from coding & agent
- GitHub launches Copilot Fusion as CPO frames AI strategy: lower the floor, raise the ceiling — mariorod1 · 2026-10-29
- TWIML Podcast: Nautilo Creator on Persistent Memory, Multi-Agent Work and Deterministic Tools — Dan_Jeffries1 · 2026-10-29
- Cut Agent Token Costs 95%: Hand Off at 50-60% Context Instead of Waiting for Compaction — tomcrawshaw01 · 2026-10-29
- GitHub Copilot SDK renamed Microsoft Copilot SDK after reaching millions of users in under a year — film_girl · 2026-10-29
- AXO: Making websites adaptive for both humans and AI agents on the same URL — Scobleizer · 2026-10-29
- exe.dev sandboxes boot from custom OCI images, opening the door to NixOS — davidcrawshaw · 2026-10-29