TestSuiteBench: coding agents kill only ~55% of bugs with generated test suites
lbeurerkellner · x · 2026-10-13
ETH researchers released TestSuiteBench: given only a natural-language spec (165 real Rust/Go/C++ apps, 5.6k-word specs, 679 reference tests each), agents must write a full test suite, scored by mutation score. Finding: frontier agents can write entire codebases, but their tests struggle to catch bugs. Claude Opus 5 (high) kills 55.5% of mutants, Opus 5.5 (medium) 53.9%, GPT 6.1 Sol 51.6%; trailing models manage only 19-29%. Validity rates of 74-97% show the tests pass — they just don't cover what matters.
Related event: TestSuiteBench: Coding Agents Fail to Write Bug-Catching Tests(2 posts)→
More from coding & agent
- Meta launches Muse Wonder and Muse Wonder Full, SOTA on agentic tasks — alexandr_wang · 2026-10-13
- SubQ Snap launches as a long-context decision model: 384K tokens, decisions in ~2-3s at $0.05/M — ryanallred · 2026-10-13
- Don't specify languages or architecture for new projects — let the model research and choose — sytelus · 2026-10-13
- GPT-6 one-shots a custom IR blaster app to replace a dead AC remote — PrasoonPratham · 2026-10-13
- NVIDIA's NURBS-based CAD kernel SMLIB goes open source, with MCP agent support — burhop · 2026-10-13
- Why checking errors is bad: Jimmy Koppel's contrarian take on defensive coding — jimmykoppel · 2026-10-13