Proposal: Benchmark Models by Cost per Completed Task, Not Token
mageblex · reddit · 2026-08-18
A user challenges the industry standard of measuring model cost by 'price per million tokens,' arguing it ignores the high hidden cost of failure. They call for new benchmarks that measure the cost per completed task across different models and providers.
More from AGI Musings
- Opinion: Second-tier AI labs may become the dumbest business, banned from frontier models — tszzl · 2026-08-18
- Shanghai AI Lab Paper: Agentic AI Poses Escalating Risks to Human Agency on Three Cognitive Levels — Shanghai-AI-Laboratory · 2026-08-18
- LLMs solving old math problems doesn't imply superiority over humans — tak3sh8 · 2026-08-18
- Two independent papers prove 1986 conjecture, both cite extensive ChatGPT use — tak3sh8 · 2026-08-18
- AGI Requires Coordination, Not Saintly Leaders to Mitigate Existential Risks — danfaggella · 2026-08-18
- AI era to create a trillion Einsteins, democratizing top-tier intelligence access — beffjezos · 2026-08-18