Prefix caching's hidden cost: editing chat history at the wrong spot costs 10x
rseroter · x · 2026-10-31
Patrick John Kelly explains prefix caching and the hidden cost of editing chat history:
- How it works: every turn resends the full conversation; providers reuse the server-side KV cache for the unchanged prefix, charging 10% of normal input cost (90% savings) with much lower latency.
- The trap: editing tokens anywhere in history shifts the position of everything after it, invalidating the cache.
- Cost of each edit type (see chart): editing the start (e.g. system prompt) → 0% cached, full 1.0x recompute; deleting a middle message → only 34% cached; pruning/appending at the tail → 90% cache retained.
- Practical rule: until model APIs expose cache controls directly, keep the start of your prompt static and put all changes at the end to keep AI systems fast and cheap.
More from coding & agent
- GitHub Copilot app is building richer dashboard widgets for personalized data views — mariorod1 · 2026-10-31
- Todoist's self-healing agent auto-fixes 1 in 5 Sentry issues and opens PRs — zeeg · 2026-10-31
- Dev's Multica 1.0 build log: 432M tokens, 89 agent runs, just $183 — jiayuan_jy · 2026-10-31
- Stardock's QA now runs agents that play the game and file reports — draginol · 2026-10-31
- Limma MCP structures unstructured data end-to-end so agents never miss a detail — FlimsyIngenuity7248 · 2026-10-31
- Agent audit metric: can you name which tests ran and files changed without reading its summary? — ronbodkin · 2026-10-31