How continuous batching keeps GPUs busy: LLM inference runs steps, not requests

arpit_bhayani · x · 2026-08-26

Arpit Bhayani explains continuous batching in LLM serving:

Generation is a loop: each iteration runs one forward pass and emits one token; a 500-token response is 500 separate steps.
Continuous batching decouples steps from requests: after every token step, the scheduler evicts finished sequences (EOS) and slots queued requests into the open slot mid-generation — token 3 of a new request can sit next to token 400 of a long one.
The GPU just runs a forward pass over whatever sequences occupy its batch slots.
This works because attention and the KV cache are per-sequence (with PagedAttention managing memory), so mixed-stage batching corrupts no one's context.

Result: high GPU utilization and lower average latency, since no request waits behind a slow one.

More from Infra