LLM Inference After Training: a walkthrough of prefill, decode and KV cache
kmeanskaran · x · 2026-09-17
A walkthrough article for readers who already know attention and transformers, aiming to make them able to trace one request through a GPU, understand the memory terms that matter, and treat FlashAttention, PagedAttention, continuous batching, and RadixAttention as named optimizations rather than buzzwords.
The author highlights a common misconception: people fluent in training narratives assume inference is just the same network minus the backward pass, which leads them to reach for optimizations before identifying which phase is slow. No CUDA needed — just Q, K, V and a trained model, starting from GPU memory terminology, with no kernel writing.
More from Infra
- Which 'token brokers' give back to open source? New data ranks upstreamed PRs to OSS inference engines — michellechen · 2026-09-17
- XeBoostLM: native C++ local LLMs on Intel NPUs and iGPUs, zero Python — Spiritual-Ad-5916 · 2026-09-17
- Chips, batteries, motors fell 99%+ in 34 years — ARK says AI is now deflating 99%+ annually — skorusARK · 2026-09-17
- MLPerf Inference v6.1 draws record 30 submitters, adds agentic inference benchmarks — TheKanter · 2026-09-17
- NVIDIA, Google and Emerald AI launch AI Energy Management Alliance for flexible data centers — dr_alphalyrae · 2026-09-17
- CoreWeave brings multi-rack NVIDIA Vera Rubin NVL72 clusters online — OnlineInference · 2026-09-17