Why inference providers must choose: fast (Groq) or cheap (CoreWeave), not both
AccBalanced · x · 2026-10-17
jessie dong lays out the core economics of inference serving: providers bill per token but pay for GPU time, forcing a binary choice between fast and cheap.
- Fast (Groq, Cerebras): few users per GPU, low latency, higher per-token cost, plus idle GPUs.
- Cheap (CoreWeave, Novita, Crusoe): much lower prices, but users wait.
It's just how GPUs work: on a Blackwell part, a solo user gets 1,000 tokens/s and the GPU produces only 1,000 tokens/s total — one person can't keep it busy. With many users sharing, total throughput hits 60,000 tokens/s, but each person sees only 20-100 tokens/s. The accompanying NVIDIA InferenceMAX chart shows gpt-oss-120b on a B200: 60,000 TPS/GPU max shared throughput vs 1,000 TPS/user max interactivity, with gains from TRT-LLM v1.0, NVSwitch parallelism and MTP.
More from Infra
- Four GB10s Wired in a Ring, No Switch — 'A Token Ring?' — l4rz · 2026-10-17
- vLLM makers Inferact partner with AMD to optimize agentic inference on Instinct GPUs — woosuk_k · 2026-10-17
- Right-sizing context windows: Gemma 4 26B voice app fits on a single H100 — Ctbhatia · 2026-10-17
- PyTorch now runs natively on AWS Trainium with zero code changes — PyTorch · 2026-10-17
- Zuck bets most inference runs locally: Meta emerging as the champion of frontier local AI — StefanoGogioso · 2026-10-17
- ModelSphere launches as open-source LLM serving platform for mixed GPU fleets — Codeblix_Ltd · 2026-10-17