The LLM inference ceiling: max tokens/sec = GPU memory bandwidth / 2N
abhijithneil · x · 2026-09-03
The author derives a first-principles estimate for LLM inference speed: producing each output token requires reading essentially every model weight from memory — for dense models there's no way around it. With N billion weights at 2 bytes each, that's 2N GB pulled per token, giving a hard ceiling of max tokens/sec = B / 2N, where B is GPU memory bandwidth. Engineering work is about approaching as high a percentage of this ceiling as possible on given hardware.
More from Infra
- LLM subscriptions compound to thousands a year — is self-hosting AI's future? — Ok_Pay1599 · 2026-09-03
- Omagrid wants to link every Omarchy machine into one giant distributed AI compute grid — dee_hw · 2026-09-03
- Nvidia to buy Hugging Face for $12.9B, grabbing the front door to open AI — The Decoder · 2026-09-03
- CPU vs GPU vs TPU vs NPU vs LPU: how 5 chip architectures trade flexibility for AI speed — Roger_M_Taylor · 2026-09-03
- llama.cpp lands three Metal MoE PRs, decode jumps from 65.6 to 73.9 tok/s — predatar · 2026-09-03
- Ex-Tesla SVP Explains How a Single Silicon Carbide Device Can Block 10,000 Volts — wandb · 2026-09-03