Tokens Are Just 0.0008% of a 400k Context: The KV Cache Math
Sauers_ · x · 2026-11-11
Sauers rebuts the claim that LLMs are fundamentally discontinuous because they emit discrete tokens: tokens are intermediate readouts from a continuous process. For a 400k context, past token identities carry 6.9M bits vs 839 billion bits in the KV cache — tokens are only 0.00082% of the information available about the past.
More from Infra
- PyTorch ships ExecuTorch WebGPU backend: run Llama 3.2 in-browser up to 7.4x faster — PyTorch · 2026-11-11
- Liquid AI models are already running in Mercedes-Benz cars, phones next — agihouse_org · 2026-11-11
- Stanford study: smart local-cloud routing cuts energy 80% and cost 73% — agihouse_org · 2026-11-11
- AGI House:本地芯片 AI 能力仅用了 10-15%,应用是最大短板 — agihouse_org · 2026-11-11
- Frontier open and closed models are only 6-12 months apart, says Stanford's Mirhoseini — agihouse_org · 2026-11-11
- Azalia Mirhoseini: model layer will commoditize, custom chips face a Cambrian explosion — agihouse_org · 2026-11-11