KV Cache Rules Everything: agent context re-reads cost you a lot, providers almost nothing
AccBalanced · x · 2026-11-21
@CompleteSkeptic published a deep explainer on KV cache economics, amplified by GergelyOrosz with his own speculative decoding deepdive: the biggest cost of coding agents is re-reading their own context, which costs providers near-zero to serve — explaining both why inference suddenly became highly profitable and why that won't last, why routing doesn't work, and how KV cache heavily constrains coding agent design.
More from coding & agent
- Why DPP picks diverse solutions: three quicksorts span a tiny volume — xtimv · 2026-11-21
- Community Notes joins MIT, UW & Stanford for WSDM Cup AI note-writing contest with $10k prizes — sanmikoyejo · 2026-11-21
- One-prompt GTA-style open world: Astra 6.0/6.2 build playable driving demo in 15–18 min — Beanyy_Weenie · 2026-11-21
- Multi-turn agentic RL on one TPU lifts Gemma 4 E2B from 9.4% to 95% win rate — Saboo_Shubham_ · 2026-11-21
- Open-source router factors prompt cache into LLM switching decisions for coding agents — Broad_Abies9390 · 2026-11-21
- StepFun unveils Step 5 Preview at SF event focused on agent reliability and cost — StepFun_ai · 2026-11-21