Six techniques that decide LLM serving performance, explained
techNmak · x · 2026-11-12
The author argues that knowing attention, KV cache and quantization isn't enough — most LLM serving performance is won outside the model itself, via six techniques:
- Prefix caching: reuse precomputed KV states for shared prompt prefixes; suffix processing and decoding run as normal.
- Continuous batching: since autoregressive requests finish at different times, the scheduler swaps finished requests for queued ones between iterations, reusing GPU capacity sooner.
- PagedAttention: KV cache grows with sequence length while final length is unknown; it splits KV states into blocks mapped to non-contiguous physical memory, fighting fragmentation and over-reservation — a memory-management technique, not an attention approximation.
- Chunked prefill: break long prompt processing into pieces so prefill and decode interleave; chunk size is a latency/throughput knob.
- Speculative decoding: a cheaper draft model proposes tokens, the target model verifies in batch; acceptance/correction preserves the target distribution, with gains depending on draft cost and acceptance rate.
- Prefill/decode disaggregation: prefill is compute-bound, decode bandwidth-bound; run them on separate worker pools and transfer the KV cache (transfer itself costs).
Key takeaway: all six leave the model unchanged — the win is in scheduling and memory management around it.
More from Infra
- Curated 9-paper reading list for LLM quantization beginners, from QLoRA to BitNet — East-Muffin-6472 · 2026-11-12
- llama.cpp PR #31915 fixes Gemma4 tool calling, user reports it "completely fixed" agentic coding — jacek2023 · 2026-11-12
- Nvidia board reportedly working with outside firm to find Jensen Huang successor — kimmonismus · 2026-11-12
- RDNA3-tuned llama.cpp fork doubles Qwen3.8-27B speed on RX 7900 XTX to ~100 tok/s — Jebbyk1 · 2026-11-12
- Ming Chi Kuo: Huang is irreplaceable to Nvidia; Rubin mass production slips to 2027 on HBM4 — zephyr_z9 · 2026-11-12
- Breakdown: Only 3% of Anthropic's 2.68M Chips Are Nvidia, Lowest in AI — teortaxesTex · 2026-11-12