TestSuiteBench: coding agents' tests catch fewer than 55% of logical bugs
OfirPress · x · 2026-10-12
TestSuiteBench shows that although frontier coding agents can write entire codebases from scratch, they struggle to generate strong test suites from specifications: their tests pass on reference implementations but catch fewer than 55% of logical mutants. A clear weak spot for agent-driven QA workflows.
More from coding & agent
- Diorama runs parallel coding agents in separate git worktrees, gongs when stuck — davidfromkansas · 2026-10-12
- poteto ships two skills that teach coding agents to verify their own work — kieranklaassen · 2026-10-12
- Indie builder ships infinite-canvas book recs site made with Cursor, Next.js and Supabase — floguo · 2026-10-12
- Manus 2.0 builds and publishes a playable 3D open-world game from a single prompt — minchoi · 2026-10-12
- Weaviate Indexes Metadata Natively: Filtered Vector Search Without a Separate Relational DB — CShorten30 · 2026-10-12
- Watching a Personal Agent 'gob' Book Japan Train Tickets End-to-End in a Cloud Browser — gabrielchua · 2026-10-12