The Inference Cost Curve is Being Misinterpreted
BlackHC · x · 2026-07-16
The article references an Epoch AI chart, but the author points out a misinterpretation: the x-axis represents "tokens per second per request," not global throughput. To make individual requests faster, servers often use smaller batches, which drives up unit costs; however, tokens actually become cheaper if the system handles a larger total volume at the same speed.
The post adds broader context: a paper indicates that inference revenue is growing about 3x per year, and models are smaller and cheaper than in 2023. The author believes this aligns much better with the real economics of inference.
Related event: BlackHC Rebuts the “AI Is the Worst Technology” Thesis(5 posts)→
More from Infra
- DeepSpeed shows AutoEP/AutoTP/AutoSP parallelism gains, hosts office hours — PyTorch · 2026-10-27
- Cloudflare Durable Objects validated as top architecture for AI agent WebSocket servers — tristanbob · 2026-10-27
- Gary Marcus: With Q3 data added, Anthropic's compute-efficiency growth shows deceleration — GaryMarcus · 2026-10-27
- Inference is just Kubernetes with a multi-trillion-dollar market, and the communities overlap — matt_slotnick · 2026-10-27
- vLLM eyes Rust migration as inference engines race to rewrite the stack — AccBalanced · 2026-10-27
- Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27