Evaluating AI Costs Requires Looking at Task Completion Rates
luckokkkk · reddit · 2026-07-15
The core point of this post is that AI costs cannot be calculated solely by token price; task completion rates and the total context consumed to actually complete a task must be factored in.
It cites Databricks' measurements: in the Claude Code scenario, an average of 742,000 context tokens are consumed per task, while Pi on Opus consumes about 236,999. The author uses this to illustrate that seemingly cheaper models or solutions may not actually be more cost-effective for the real cost of "completing the same task."
This type of comparison leans closer to "task-level cost" rather than "token-level cost," which is crucial for evaluating the economics of coding assistants, agentic workflows, and inference services.
Related event: True AI Cost Depends on Task Completion Rates(2 posts)→
More from Infra
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11
- PyTorch Day Korea 2026 launches first offline conf, CFP closes Sept 13 — PyTorch · 2026-09-11
- Local LLM server dilemma: 4x CMP-170HX (price up 53% in 20 days) vs Mac Studio M5 Ultra — rumboll · 2026-09-11
- llama.cpp lands Flash Attention tuning for RDNA4, big prefill gains on AMD — pmttyji · 2026-09-11