Cached tokens explained: how prefix caching cuts agent costs to 2% and what breaks the hit rate
Arindam_1729 · x · 2026-10-28
Arindam's article explains prefix caching for agent loops: every turn resends the system prompt, tool definitions and history. On Nebius, DeepSeek V4.1 Flash 1017 uncached input costs $0.30/1M tokens vs $0.006/1M cached — 2% of list price. With illustrated diagrams, it covers five ways to keep the prompt prefix identical (order, tools, history append-only, timing, one model per conversation), why caching is best-effort with no SLA (cachedtokens 0 can mean eviction, different replica, or model update), serverless caches being shared per model rather than org-isolated, and how to budget at list price while treating hits as upside.
More from coding & agent
- 8 open-source repos to self-host a lean AI agent stack instead of stacking $20/month SaaS — SaiTarunRonanki · 2026-10-28
- Single Linode + Google's open-source ax runtime orchestrates agent fleet to build 300 buildings — rakyll · 2026-10-28
- Dev builds mcp-warden so AI agents can request actions instead of holding plaintext API keys — Relevant_Mousse_9117 · 2026-10-28
- Qwen Code v0.26.3 ships workflow mode, memory cadence and compaction fixes — qwen-code-review-bot · 2026-10-28
- Dev gives System 1 decision models virtual gamepads in a Godot AI fighting tournament — ViceGodIm · 2026-10-28
- Dev builds a personalized Hacker News digest with Grok Bot every morning — seanmcdonaldxyz · 2026-10-28