Same AI agent scores 97% or 39% depending on how you define success
hugobowne · x · 2026-09-21
Hugo Bowne-Anderson illustrates how eval definitions change everything, using an Anthropic agent example:
- 73% success on a single attempt
- 97% chance of at least one success across three independent attempts
- 39% if all three attempts must succeed
Same agent, three very different numbers. If you can check multiple proposed fixes and keep the passing one, success-within-a-few-attempts is useful; if customers rely on the agent every time, consistency is what matters.
He's teaching a free Lightning Lesson, "AI Agent Evals: Test What Matters for Your Agent" (Sep 21/22), covering turning failures into eval cases, choosing between code checks, LLM judges and human review, and verifying that changes to your agent actually help.
Related event: Agent eval success rates hinge on how you define success(2 posts)→
More from coding & agent
- Dev to open-source an Obsidian Agent plugin and one-prompt website deployer — vista8 · 2026-09-21
- JEV eval: when state evidence is ambiguous, choice order dominates and IDK fails 0/80 — iamrobotbear · 2026-09-21
- Repowise Indexes Codebases for AI Agents via MCP, Hits 6.7k GitHub Stars — tom_doerr · 2026-09-21
- Reverse-engineering Jev: its architecture, philosophy, and where it falls down — iamrobotbear · 2026-09-21
- Veteran dev: AI built a full MMO in 2 hours, so what am I for? — devontec · 2026-09-21
- Workflow tip: dictate rambling prompts to ChatGPT, then paste into your agents — athyuttamre · 2026-09-21