KV Cache Math From Scratch: A Deep Workshop on LLM Inference at Scale
AI Engineer · youtube · 2026-09-08
A long-form workshop by Audible senior engineer Harshul Jain and independent researcher Tanmay Sah breaks down LLM inference cost and optimization from first principles.
Key number: one token of KV cache on Mistral 7B costs 131 KB; at 16k context and 80 concurrent users, the cache alone wants 42 GB of GPU memory — why requests fail on a 24 GB card.
Three pain points: memory scaling with context, TTFT degrading with longer prompts, and throughput collapse from naive serial serving.
Model-side: quantization, multi-head → multi-query → grouped-query → latent attention, flash attention and tiling.
Serving-side: paged attention (borrowed from OS paging), continuous batching, prefix caching, KV quantization — each benchmarked against a plain baseline.
Engine choice: no statistical difference between vLLM and SGLang on standard workloads, but a 3–4x gap once agentic branching enters the picture. Slides and runnable notebooks are in the repo.
More from Infra
- MagicaLabs says $4M pretraining beat all public base models, 50x more compute-efficient than DeepSeek — magicailabs · 2026-09-09
- Qualcomm confirms AWS deal is baked into its $15B FY29 data center revenue target — BenBajarin · 2026-09-08
- Data center construction spend jumps $25B in six months, job openings top 300k — AccBalanced · 2026-09-08
- CDOs backed by GPU leases likely coming as compute financialization accelerates — AccBalanced · 2026-09-08
- Citi TMT takeaways: hyperscaler backlog near $1.7T, AI capex tracking to $3.8T by 2030 — sanjaykalra · 2026-09-08
- AI data center interconnect chip startup Celero raises $275M at $3B+ valuation — dinabass · 2026-09-08