Tsinghua's cache engineering for agents cuts cost 27.7% and boosts P95 latency 1.5×

kalyan_kpl · x · 2026-10-22

A Tsinghua/CUHK paper, "Cache Engineering for Agents," tackles KV cache I/O and storage becoming the dominant cost in long LLM agent trajectories:

Prior context engineering trims or summarizes history, but modifying the trajectory prefix invalidates cached states and triggers costly re-prefilling, and retaining too little context hurts accuracy;
Cache engineering lets agents directly control their active KV cache: proactively dropping selected content to free future attention and repositioning retained keys' encodings — without touching the trajectory or re-prefilling;
The paper formalizes cache operation sequences, showing identical token sequences yield different cache prefixes under different operations, breaking prefix matching — hence the co-designed AKV serving engine, with a stateless interface compatible with existing serving APIs while managing stateful cache reuse, eviction and restoration;
Results: a tool-response rolling-drop policy lifts average benchmark scores by 1.7 points and cuts average cost 27.7%; AKV achieves up to 1.2× output throughput and 1.5× faster P95 latency; recursive self-improvement of cache policies saves another 7.1%.

Code: github.com/thu-nics/AKV.

More from coding & agent