After 19 full runs, this dev ditched 100% as the bar for coding-agent evals
oliver-zehentleitner · reddit · 2026-09-18
The author runs an 88-case eval suite for a coding-agent skill (Keep the Why): each case runs a real agent session on a fixture project, graded by an LLM judge. The goal of three consecutive 88/88 runs proved futile over 19 full runs.
- Overall pass rate hit 98.6%, but roughly one case failed per run—usually a different one each time, often cases that had passed 20-30 times before
- Neither the agent nor the judge can be seeded, so "three perfect runs" measures luck, not skill quality
- Chasing isolated failures made things worse: 3 of the added fix-sentences flipped neighboring cases that had never failed
The new release criterion: every case passes at least 2 of 3 runs, and no single run has more than one failure. That gives failures meaning—isolated failure is sampling noise, the same case failing twice points to wording, multiple failures in one run signals a regression. Version 0.17.0 scored 87/88, 88/88, 87/88, with 86 cases passing all three runs.
A side win: six prose rules controlling confirmation behavior were replaced by a 6x3 decision table—21% shorter with zero decision flips across two full eval series. Takeaway: when both the system under test and the evaluator are sampled, requiring repeated perfection can be a bad acceptance criterion.
More from coding & agent
- Tip: Have Your AI Agent Classify PR Merge Risk by Reversibility and Blast Radius — mattpocockuk · 2026-09-18
- Anthropic Engineer's 2-Hour Workshop: Replacing Agent Loops with Graph Engineering — JohnAlexander · 2026-09-18
- HF CLI cheat sheet: one-liners to trace model lineage, datasets and papers — vanstriendaniel · 2026-09-18
- User burns through ChatGPT Pro weekly quota in 2 days running Codex — craigbalding · 2026-09-18
- Investor: QNX's microkernel is an undervalued moat for the agentic AI era — pdamodaran · 2026-09-18
- Awesome Code-as-X: curated papers unifying agent research via programs as representations — ccloy · 2026-09-18