Tsinghua paper: letting agents manage their own KV cache cuts cost by 27.7%
dair_ai · x · 2026-10-22
A Tsinghua/CUHK paper introduces "cache engineering": instead of trimming or summarizing trajectory text (which invalidates KV cache prefixes and triggers costly re-prefills), agents get programmable operations to drop selected cache from future attention and reposition retained keys. With KV cache reads accounting for up to 82.4% of API cost, a rolling drop policy for old tool responses improves benchmark scores by 1.7 points while cutting average cost by 27.7%. The co-designed KV serving engine delivers up to 1.2× output throughput and 1.5× P95 latency gains, and recursive self-improvement of cache policies saves another 7.1% over expert design. Code: github.com/thu-nics/AKV
More from coding & agent
- YC F26 startup Latent reads model internals to catch hallucinations at 530x lower cost — ycombinator · 2026-10-22
- We need an L0.5 for continual learning: agent state stacks up in 4 layers — willcb · 2026-10-22
- DuckDB + Hugging Face Amsterdam workshop: build an AI feedback pipeline over 2.1M Amazon reviews — lhoestq · 2026-10-22
- Veracode report: AI now writes half of committed code but only 56% pass security checks — WeldPond · 2026-10-22
- Will Brown: Continual Learning in the Harness Is Like a Computer Without RAM — willcb · 2026-10-22
- Migrating an entire Hermes automation workload to Antigravity took about 2 hours — rseroter · 2026-10-22