Same AI agent scores 97% or 39% depending on how you define success
hugobowne · x · 2026-09-21
Hugo Bowne-Anderson illustrates how eval definitions change everything, using an Anthropic agent example:
- 73% success on a single attempt
- 97% chance of at least one success across three independent attempts
- 39% if all three attempts must succeed
Same agent, three very different numbers. If you can check multiple proposed fixes and keep the passing one, success-within-a-few-attempts is useful; if customers rely on the agent every time, consistency is what matters.
He's teaching a free Lightning Lesson, "AI Agent Evals: Test What Matters for Your Agent" (Sep 21/22), covering turning failures into eval cases, choosing between code checks, LLM judges and human review, and verifying that changes to your agent actually help.
Related event: Agent eval success rates hinge on how you define success(2 posts)→
More from coding & agent
- User automates entire video clipping workflow with Nous Research's Hermes agent — Teknium · 2026-11-12
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Prime agent ships with day-one multi-agent communication, runs in Minecraft unmodified — sarahcat21 · 2026-11-12
- Autonomous agent edits just 1,024 weights to lift Qwen3.5 0.8B from 7.4% to 37.4% on MATH-500 — VictorSanh · 2026-11-12
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Coding harnesses hand shell access to inference providers — 4-step gateway mitigation — shensi · 2026-11-12