Measuring an LLM agent that writes tests before the code exists: 33/33 planted bugs caught
Infinite-Fly-2756 · reddit · 2026-11-10
The author open-sourced Agent QA Kit (MIT), a QA workflow where a Claude Sonnet agent via Claude Code writes one unit test and one browser test per acceptance criterion from the ticket, before the feature exists. Each test must fail first; a human approves them once, then they're frozen. When the feature ships, a Claude Haiku agent runs them, with plain scripts verifying verdicts against actual test reports — agents never decide alone.
Validation: 17 small tickets on a demo shop with hidden tests and planted bugs, 3-6 trials each on an isolated CI runner. The latest version passed 21/21 trials on 6 tickets and caught all 33 planted bugs; the prior version passed 49/51 across all 17.
Surprising findings:
- Asking the agent to write more test cases made things worse overall: it caught a previously missed bug in 8/15 trials, but false failures on correct code rose (5 in 48 trials vs 2), mostly from miscalculated expected values. Asking it to list what its tests don't cover worked better.
- Testing nonexistent code, it guesses markup (assumed a "textbox" for a search field; expected totals to appear alone, not as "Total €9.40"). Two rules fixed it: find fields by label, check values as part of element text.
- Every refused tool call costs a turn; refusal messages that say what to do instead dropped refusals from 16 to 1 and saved 2 turns per ticket.
- Weakest link was human: on two real tickets, tests were merged before anyone approved them.
Small samples — signals, not proof. Method and raw numbers are in the repo.
More from coding & agent
- Amazon swaps Sonnet 4.6 for a decision model in ticket labeling, cutting cost by 98% — victormustar · 2026-11-10
- 360-trial benchmark finds code knowledge graphs give coding agents no clear accuracy gain — JeremyCMorgan · 2026-11-10
- Dev claims a $25/month Claude Code Polymarket bot scanned 14,000 wallets in 4 minutes — Aiden_Tech_Ai · 2026-11-10
- Jev Engineering's dedicated micro-decision model vs Flash subagents in Antigravity — Pokeasss · 2026-11-10
- Builder tip: give your custom AI tooling a way to calibrate responses with user feedback — menhguin · 2026-11-10
- Async agents that actually work: watchlists, autonomy levels, and context — menhguin · 2026-11-10