Prefill vs Decode Explained: Parallel Prompt Processing Fills the KV Cache
cfregly · x · 2026-11-11
The cartoon series explains LLM inference's two phases: Alice as prefill processes all prompt tokens in parallel to fill the KV cache, while Lori as decode reuses and extends that cache as Jensen generates his reply one token at a time ("I… can… explain."). Links NVIDIA's inference optimization blog covering KV cache memory growth, PagedAttention, quantization, and in-flight batching.
Related event: Illustrated Guide to LLM Inference Phases and CUDA Error Pitfalls(2 posts)→
More from Infra
- Commerce Dept delays entity list decision, Megaspeed can still buy advanced Nvidia AI chips — GarrisonLovely · 2026-11-11
- Liquid AI models are already running in Mercedes-Benz cars, phones next — agihouse_org · 2026-11-11
- AGI House: Only 10-15% of On-Device Chip AI Power Used, Apps Lag Behind — agihouse_org · 2026-11-11
- AWS's agent evaluation playbook: built-in, DeepEval and AutoEval on Bedrock AgentCore — AWS ML Blog · 2026-11-11
- AMD MI355X posts fastest single-user inference, ~2.4x H200 tokens per dollar in tests — AryHHAry · 2026-11-11
- HF tokenizers v1.0.0-rc.4 restores Python APIs and encodes up to 17x faster — art_zucker · 2026-11-11