The LLM inference ceiling: max tokens/sec = GPU memory bandwidth / 2N
abhijithneil · x · 2026-09-03
The author derives a first-principles estimate for LLM inference speed: producing each output token requires reading essentially every model weight from memory — for dense models there's no way around it. With N billion weights at 2 bytes each, that's 2N GB pulled per token, giving a hard ceiling of max tokens/sec = B / 2N, where B is GPU memory bandwidth. Engineering work is about approaching as high a percentage of this ceiling as possible on given hardware.
Related event: Memory Bandwidth Caps LLM Decoding: A Simple Token/s Formula(2 posts)→
More from Infra
- DeepSeek's token density yields ~$38/GPU-hr revenue vs $3.50 for raw IaaS, engineer estimates — bookwormengr · 2026-10-21
- Sizing a 1 GW datacenter: ~522,000 Blackwell GPUs fit inside — bookwormengr · 2026-10-21
- At 85% billable uptime, a 1 GW fleet supplies 3.89B GPU-hours/year — bookwormengr · 2026-10-21
- DeepSeek's production density reportedly hits ~$38 per GPU-hour at saturation — bookwormengr · 2026-10-21
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- Indie dev replaces $1k OpenAI bill with a YAML pipeline on spot GPUs and vLLM — cmm324 · 2026-10-21