A Year of LLM Inference Optimization: vLLM vs SGLang, Load Tests to 2000 rpm
abhijithneil · x · 2026-09-03
The author recounts a year spent making LLM inference faster: ablations on single-GPU boxes, multi-node fleets, benchmarking vLLM against SGLang, and load tests ramping to 2000 rpm before falling over at minute six.
His biggest lesson: he lacked a mental model of the machine, so every flag was a coin flip. He starts from first principles — for a dense model, the GPU reads essentially every weight from memory per output token, so N billion weights at 2 bytes each means 2N GB pulled per token.
A solid read for anyone who wants to reason about the inference stack rather than blindly copy configs.
Related event: A Year of LLM Inference Tuning: vLLM vs SGLang(2 posts)→
More from Infra
- Data center developers are pouring into Iceland for geothermal energy and cold climate — Polymarket · 2026-09-03
- Teacher With 16GB VRAM Hits a Wall: Local LLMs Keep Failing at MCP Tool Use — whakahere · 2026-09-03
- Kimi K3 Draft Collection released: EAGLE-3, DFlash2 and DSpark draft models trained on GB200 — hongyangzh · 2026-09-03
- One bag of almonds' 'waste water' could power 100 ChatGPT queries a day for 385 years — Polymarket · 2026-09-03
- Open-source Marin kicks off 535B/23B MoE pretraining run, fully documented in public — dlwh · 2026-09-03
- Agents spend most of their life on CPU work, not GPU token generation — ai · 2026-09-03