One Blackwell GPU trade-off: 60,000 TPS shared vs 1,000 TPS for a single user
FinanceYF5 · x · 2026-10-25
Inference providers can't be both fast and cheap—it's fundamentally a resource allocation problem on the same GPU.
- Serving speed requires fewer users per GPU, idling compute and raising token costs: Groq and Cerebras choose speed; CoreWeave, Novita and Crusoe lean cheap.
- The chart shows NVIDIA's gpt-oss-120b on a Blackwell B200: a single user gets 1,000 TPS, while sharing pushes total throughput to 60,000 TPS/GPU but drops each user to 20–100 TPS.
- Latest optimizations (TRT-LLM v1.0, NVSwitch parallelism, Multi-Token Prediction) deliver roughly 5x throughput gains.
More from Infra
- ByteDance, Alibaba and Tencent order 3M+ Nvidia RTX PRO 5500 cards worth $35B+ — Miles_Brundage · 2026-10-25
- Nebius hits 67B tokens/day on OpenRouter, but rivals process up to 2T daily — AccBalanced · 2026-10-25
- OpenRouter analysis finds evidence of spoofing-style price manipulation by inference providers — AccBalanced · 2026-10-25
- EmbeddedLLM receives AMD MI350X server donation to advance vLLM ROCm work — AccBalanced · 2026-10-25
- 106 Tesla Megapacks to replace a former Belgian coal plant with 400 MWh storage — XFreeze · 2026-10-25
- Running Qwen 3.8 Flash Next on 2x CMP 170HX mining GPUs, config shared — leonbollerup · 2026-10-25