Continuous batching explained: the scheduler behind vLLM's 23x throughput
blaizedsouza · x · 2026-09-15
A clear breakdown of continuous batching, the scheduling technique behind vLLM's 23x throughput jump and now the default in every serving engine:
- Traditional ML inference: a batch is a padded matrix; one forward pass, all rows finish together.
- LLM decoding differs: each pass yields one token per sequence, and output length is unknown until the stop token.
- Static batching waste: a 30-token request holds its slot until the slowest 400-token request finishes, while GPU weight reads cost the same regardless.
- Continuous batching moves the scheduling boundary to every token step, letting finished requests exit and new ones join immediately.
The linked article also covers internals like token budgets, KV allocation, and preemption.
More from Infra
- Your AI agent would cost 260% more without prompt caching, Arena data shows — arena · 2026-10-30
- Nebius runs auction-driven GPU pricing, lifting rates ~20% amid tight supply to 2028 — BenBajarin · 2026-10-30
- macOS update silently re-adds eMudhra root certs users had removed — steipete · 2026-10-30
- Bittensor's Terza Inference routes one API across miners' models on subnet 103 — const_reborn · 2026-10-30
- SGLang Summit 2026 sells out sponsor booths and Full Passes ahead of Nov 12–13 event — ying11231 · 2026-10-30
- OpenRouter rebuilds provider routing: 46% fewer capacity errors, 16.9% faster TTFT — steipete · 2026-10-30