Tsinghua & CUHK propose 'Cache Engineering': agents edit KV cache without re-prefilling

TheTuringPost · x · 2026-10-28

A paper from Tsinghua University and CUHK introduces Cache Engineering for Agents, going beyond context engineering for long agent trajectories.

Problem

KV cache I/O and storage dominate cost as agent trajectories grow, which can also exceed max context length.
Existing context engineering (trimming/summarizing history) modifies trajectory prefixes, invalidating cached states and triggering costly re-prefilling; retaining too little context hurts accuracy.

Approach

Let agents directly control their active KV cache without touching the trajectory or re-prefilling: a drop operation excludes selected cache from future attention, and repos adjusts position encodings of retained keys.
They formalize these operations and identify history dependence as a serving challenge: identical token sequences yield different cache states under different operation sequences, so prefix matching is insufficient for correct cache reuse.

System & results

The co-designed AKV serving engine exposes a stateless interface compatible with existing serving APIs while internally handling stateful cache reuse, eviction, and restoration.
A tool-response rolling-drop cache policy improves average benchmark performance.

Paper: arXiv 2610.28577, code available on GitHub.

More from coding & agent