The cheaper agent run can cost more per usable result, argues new analysis
expectationsledger · reddit · 2026-09-17
A Reddit post uses a deliberately made-up example to show why API bills alone mislead when comparing agents:
- Agent A: $10 for 100 tasks, 20 pass acceptance — $0.50 per accepted result
- Agent B: $30, 90 pass — about $0.33 per accepted result
The lower bill hides a worse per-usable-result cost. The author proposes reporting three metrics together: total model/tool spend including retries, acceptance rate against a test fixed in advance, and human minutes spent reviewing and fixing output. Systems that only handle easy cases look cheap while handing expensive work back to humans. Only when acceptance rates and review effort are equal can the lower bill be judged the better outcome.
More from coding & agent
- The harness tax: Claude Code costs 2x Pi at the same 15.3 turns, with 10x initial context on SWE-bench Lite — HamelHusain · 2026-09-17
- Jev: an open-source action-picker that splits agent thinking from clicking — alexcovo_eth · 2026-09-17
- CROA open-sources a deterministic execution layer enforcing trajectory-level constraints on AI agents — CROA_PROJECT · 2026-09-17
- Moving coding agent memory off-machine: from Dropbox CONTEXT.md to MCP shared memory — Asly97 · 2026-09-17
- What's the worst 'fix' you've seen an AI coding agent make? — fromkrish · 2026-09-17
- Self-taught builder shipped 900 PRs in 3 months running 18 terminal agents to automate insurance brokerage — CShorten30 · 2026-09-17