6 serving-side techniques that make LLM inference faster - from prefix caching to PD disaggregation
techNmak · x · 2026-09-23
A detailed thread explains how much LLM serving performance comes from around the model, not the model itself, breaking down six techniques that each attack a different source of waste:
- Prefix caching: reuse KV states for shared prefixes instead of re-running prefill.
- Continuous batching: iteration-level scheduling decouples GPU capacity from fixed request groups.
- PagedAttention: block-based, non-contiguous KV allocation to eliminate over-reservation and fragmentation - fundamentally a memory management idea.
- Chunked prefill: split long prompts so prefill mixes with decode work, trading reduced stalls for chunking overhead.
- Speculative decoding: a cheap draft model proposes tokens that the target model verifies in batch, preserving the sampling distribution; only pays off when saved target steps exceed drafting costs.
- Prefill/decode disaggregation: prefill is compute-bound, decode bandwidth-bound, so systems like DistServe and Splitwise run them on separate worker pools.
More from Infra
- Qualcomm's new chips stream 30B MoE weights from flash to RAM on demand — ryanshrout · 2026-09-23
- Qualcomm says Snapdragon's Hexagon NPU now runs 30B+ parameter MoE models on-device — samcharrington · 2026-09-23
- OpenAI Upgrades GPT-6 Prompt Caching With Higher Hit Rates and New Diagnostics — OpenAI News · 2026-09-23
- NCCL 2.31.2 ships GPU-driven CFT/RMA, 0-SM collectives, better multi-NIC for Blackwell-scale training — SkyLi0n · 2026-09-23
- vLLM ROCm lead maintainers only recently got persistent access to an MI355X cluster — AccBalanced · 2026-09-23
- South African Rights Groups Demand Moratorium on US Big Tech Data Centers — ChinasaTOkolo · 2026-09-23