KV Cache Math From Scratch: A Deep Workshop on LLM Inference at Scale
AI Engineer · youtube · 2026-09-08
A long-form workshop by Audible senior engineer Harshul Jain and independent researcher Tanmay Sah breaks down LLM inference cost and optimization from first principles.
Key number: one token of KV cache on Mistral 7B costs 131 KB; at 16k context and 80 concurrent users, the cache alone wants 42 GB of GPU memory — why requests fail on a 24 GB card.
Three pain points: memory scaling with context, TTFT degrading with longer prompts, and throughput collapse from naive serial serving.
Model-side: quantization, multi-head → multi-query → grouped-query → latent attention, flash attention and tiling.
Serving-side: paged attention (borrowed from OS paging), continuous batching, prefix caching, KV quantization — each benchmarked against a plain baseline.
Engine choice: no statistical difference between vLLM and SGLang on standard workloads, but a 3–4x gap once agentic branching enters the picture. Slides and runnable notebooks are in the repo.
More from Infra
- Nearly half of Claude revenue now flows through cloud marketplace commitments, up from 11% — BenBajarin · 2026-10-26
- Anthropic ARR goes vertical after price cuts: tokens up 4x, $100B+ ARR back in play — GavinSBaker · 2026-10-26
- Alex Banks argues power caps GPU supply, real risk is demand cooling first — thealexbanks · 2026-10-26
- Analyst: Ant has the most fungible AI workload of any model provider today — BenBajarin · 2026-10-26
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26