A year of LLM inference tuning: vLLM vs SGLang, 2000-rpm load tests, and the real bottleneck
abhijithneil · x · 2026-09-03
The author recounts a year spent speeding up LLM inference: ablations on single-GPU boxes and multi-node fleets, benchmarking vLLM against SGLang, and load tests that ramped to 2000 rpm before falling over at minute six. His biggest lesson: the real bottleneck was lacking a mental model of the machine, making every flag feel like a coin flip. The thread will expand into concrete inference optimization methodology.
Related event: A Year of LLM Inference Tuning: vLLM vs SGLang(2 posts)→
More from Infra
- LLM subscriptions compound to thousands a year — is self-hosting AI's future? — Ok_Pay1599 · 2026-09-03
- Omagrid wants to link every Omarchy machine into one giant distributed AI compute grid — dee_hw · 2026-09-03
- Nvidia to buy Hugging Face for $12.9B, grabbing the front door to open AI — The Decoder · 2026-09-03
- CPU vs GPU vs TPU vs NPU vs LPU: how 5 chip architectures trade flexibility for AI speed — Roger_M_Taylor · 2026-09-03
- llama.cpp lands three Metal MoE PRs, decode jumps from 65.6 to 73.9 tok/s — predatar · 2026-09-03
- Ex-Tesla SVP Explains How a Single Silicon Carbide Device Can Block 10,000 Volts — wandb · 2026-09-03