Continuous batching explained: the scheduler behind vLLM's 23x throughput
blaizedsouza · x · 2026-09-15
A clear breakdown of continuous batching, the scheduling technique behind vLLM's 23x throughput jump and now the default in every serving engine:
- Traditional ML inference: a batch is a padded matrix; one forward pass, all rows finish together.
- LLM decoding differs: each pass yields one token per sequence, and output length is unknown until the stop token.
- Static batching waste: a 30-token request holds its slot until the slowest 400-token request finishes, while GPU weight reads cost the same regardless.
- Continuous batching moves the scheduling boundary to every token step, letting finished requests exit and new ones join immediately.
The linked article also covers internals like token budgets, KV allocation, and preemption.
More from Infra
- NVIDIA: full-stack NIM tuning delivers 2.5x more concurrent users on Nemotron 3 Ultra — NVIDIAAI · 2026-09-15
- How Much Does Local LLM Inference Really Cost? A Dev Added an Electricity Calculator — giveen · 2026-09-15
- Hugging Face Rounds Up Which Open LLMs Are Best for On-Device Inference — NielsRogge · 2026-09-15
- Single Pure-C99 Inference Engine Runs Both BitNet Ternary and GGUF, No Python or CUDA — shifu_legend · 2026-09-15
- Dev weighs ChatGPT subscription via OAuth vs API pricing for a production RAG app — builtforoutput · 2026-09-15
- Cognichip launches ACI Enterprise: one engineer finishes chip front-end design in 10 days — kimmonismus · 2026-09-15