LLM inference flips between compute-bound and memory-bound within a single request
techNmak · x · 2026-11-07
A detailed thread (with infographic) on LLM serving: prefill is usually compute-limited while low-batch decode is memory-bandwidth-limited. For a 32-layer GQA model with 8 KV heads (BF16), an 8,192-token sequence needs 1 GiB of KV cache — 16 sequences already cost 16 GiB. Covers continuous batching, chunked prefill, PagedAttention, prefix caching, FlashAttention, speculative decoding and prefill/decode pool separation. Key takeaway: measure TTFT, inter-token latency, tail latency, KV utilization and cost under realistic workloads, not just raw tokens/s.
More from Infra
- Data centers' water use is just ~4% of US golf course irrigation, study argues — kimmonismus · 2026-11-07
- Data centers' water use vastly overstated: just 4% of US golf course irrigation — kimmonismus · 2026-11-07
- ByteDance reportedly weighing major expansion of AI data centers in Brazil — Polymarket · 2026-11-07
- Download an open-source model this weekend: one 5-20GB file, runs offline in ~20 min — alex_verem · 2026-11-07
- Underdog Woof: a 31.7 MB TTS model that runs on CPU 13.8x faster than real time — victormustar · 2026-11-07
- Four TTS models small enough to run on CPU, smallest just 1.4M params — vanstriendaniel · 2026-11-07