Tsinghua & CUHK propose cache engineering: 27.7% cost cut and +1.7 benchmark for agents
TheTuringPost · x · 2026-10-28
Context engineering can backfire: trimming or summarizing older messages forces KV-cache recomputation and increases compute. Tsinghua and CUHK propose cache engineering instead—keep the text history intact and edit cache availability via two ops alongside append: "drop" (removes KV entries from future attention) and "repos" (adjusts positional encodings). They built AKV, a serving engine managing cache reuse, eviction, and restoration with a stateless API, an extended radix tree tracking operation histories, and segment-batch attention for single-pass restoration. A simple policy that gradually drops older tool responses cut average cost 27.7%, added +1.7 benchmark points, and delivered 1.18× throughput / 1.51× P95 latency; letting agents refine their own cache policies saved another 7.1%. Caveat: it requires specialized serving infrastructure.
Related event: Tsinghua and CUHK Propose Cache Engineering to Cut Agent Costs(2 posts)→
More from Infra
- Row-Bot: a local 27B model runs a shop's morning ops entirely on one PC via MCP — Acceptable-Object390 · 2026-10-28
- Streaming inference stack runs full-precision DeepSeek V4.1 on one 5090 at 1770 tok/s prefill — Front_Eagle739 · 2026-10-28
- Richard Socher's Recursive open-sources Flint: 40B MoE, 2.5B active, Apache 2, runs coding locally on Mac — altryne · 2026-10-28
- Cantor: SK hynix trades at just 3x CY27 EPS as AI cycle proves durable — firstadopter · 2026-10-28
- vLLM Semantic Router and AMD release Vela 3.0: 380M model beats GPT-5-mini at zero-shot — vllm_project · 2026-10-28
- Wall Street gripped by new anxiety as it bankrolls AI's vast data center buildout — shiringhaffary · 2026-10-28