Agentic Cache Engineering: Agents Managing Their Own KV Cache Cut Costs 7.1% Further
mrdrozdov · x · 2026-10-22
dairai highlighted a paper on KV cache management for long-running agents, with Rulin Shao adding extension ideas.
Problem: When an agent summarizes or trims its history, prefix changes invalidate all cached KV entries after it, forcing expensive re-prefill of retained context.
Approach — cache engineering: The agent gets two primitives to operate on its own KV cache: one removes entries from future attention, the other shifts positions of remaining keys. The trajectory text stays unchanged, preserving cache across context edits.
Results: Recursive self-improvement applied to cache policies cuts cost by another 7.1% over expert-designed policies.
Discussion: Shao proposes adding an LLM request argument so agents can tell the serving system when to re-prefill the suffix cache when context edits introduce conflicting information — calling cache-space context management an exciting optimization space.
More from coding & agent
- Hermes Agent demo: rigged, dancing game characters from one prompt each, no Blender — alexcovo_eth · 2026-10-22
- Hands-on with EverOS Cloud: persistent agent memory that survives fresh sessions — nikola_mr64990 · 2026-10-22
- Setting up a Grok bot routine to inventory and rotate your API keys across devices — alexcovo_eth · 2026-10-22
- Claude Code built a real-time browser city destruction demo with 240 unit tests — smith2008 · 2026-10-22
- PSMcpServer turns PowerShell scripts into MCP tools just by dropping them in a folder — dfinke · 2026-10-22
- Developer slams MCP: like GraphQL, too open-ended, hard to cache and easy to attack — ZeroCool86 · 2026-10-22