Agent Evals Should Score Outcome and Reliability Separately, Dev Argues
PromptPhanter · reddit · 2026-09-09
The author argues agent benchmarks wrongly collapse everything into a single pass/fail rate, hiding two distinct failure modes: an agent can be operationally reliable yet do the wrong task, or accomplish the task despite provider errors and retries.
Proposed split:
- Outcome: did the agent actually accomplish the user's goal?
- Reliability: did the session reach a structurally usable completion without unrecovered operational failure?
Recovered errors should stay visible but be accounted in wasted spend and critical-path time, not as fractional reliability failures. The author also stresses verifying external state — 'no exception' is not success if the agent claims an email was sent that doesn't exist.
More from coding & agent
- Smart LLM routing cuts costs 69% on 120 tasks while keeping 99.2% success rate — shensi · 2026-09-09
- Test-time adaptation via human-AI interaction paper open-sources full codebase — dan_fried · 2026-09-09
- Users find Google Astra cheaper at higher reasoning levels — brandon_galang · 2026-09-09
- t3code hands-on: manage Claude/Codex/Grok sessions from your phone — kamalgupta09 · 2026-09-09
- "No-Model-in-the-Loop" MCP retrieval layer cuts coding agent tool calls by 28% — Cryvixx · 2026-09-09
- Lovable's First Engineer Emil Ahlback Exits to Build getenergy with Friends — emilahlback · 2026-09-09