KV, prefix, prompt, semantic: the four LLM caches explained, and where each saves compute
blaizedsouza · x · 2026-10-25
A well-crafted explainer thread breaks down the four caching mechanisms in LLM inference and where each saves repeated work:
- Context: agents resend the same system instructions, tool definitions, and history every turn; the initial processing (prefill) drives first-token latency.
- KV caching: during generation, keys/values used by attention are stored and reused per new token, trading memory for compute.
- Prefix caching: across requests, servers reuse saved attention state when a request begins with an already-processed sequence, skipping prefill.
- Prompt caching: vendor-level explicit caches that discount and accelerate fixed context prefixes.
- Semantic caching: semantically similar queries return cached answers directly.
The author also published a companion article on KV caching, explaining why the first token from ChatGPT/Claude is noticeably slower while the rest stream instantly — deliberate engineering, not a UI quirk.
More from Infra
- 8 mixed GPUs plus 256GB DDR4 vs DGX Spark: is this MoE inference rig viable? — Ambitious_Fold_2874 · 2026-10-25
- DeepMind paper: 5-second-per-layer quadratic simulations predict LLM training spikes — burny_tech · 2026-10-25
- Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale — blaizedsouza · 2026-10-25
- Memory is 70-80% of AI rack silicon, and Rasgon asks what if prices never collapse — firstadopter · 2026-10-25
- GLM 5.3 Flash NVFP4 local-run repo assembled, tested up to 1M context (512K recommended) — HankYeomans · 2026-10-25
- Agents riding NVIDIA GPUs to unlock more NVIDIA GPUs: an Ouroboros moment — generativist · 2026-10-25