Cache engineering beats context engineering: Tsinghua's AKV cuts agent inference cost 27.7%
机器之心 · wechat · 2026-11-17
Tsinghua and CUHK researchers propose "cache engineering" and open-source an inference system, AKV (built on SGLang): instead of compressing an agent's history via truncation or summarization, agents directly manage the KV cache, cutting cost and latency while improving success rates.
- Problem: editing context text invalidates downstream KV cache, forcing costly re-prefill. On BrowseComp-Plus, keeping only the last 12 tool results actually costs 1.85x the full context
- Method: keep all history, expose three cache operations (Append / Drop / Repos for positional re-encoding); an extended radix tree handles cache-state matching across different operation orders
- Results: across 3 models and 4 agent benchmarks, AKV cuts average inference cost 27.7% and raises success rate 1.7pp vs Standard; with GPT-OSS-120B throughput reaches 1.18x and P95 latency drops by roughly a third
- RSI experiments let the agent itself search and iterate cache policies, cutting cost another 7.1% beyond the hand-designed policy
More from coding & agent
- REA-Video skill lets Claude reverse-engineer video editing styles; rival claims 20x speed at SSIM 0.96 — toolstelegraph · 2026-11-17
- Developer retries Amp Code after months and finally gets the hype: "completely different" — g3off_who · 2026-11-17
- Dev shares 5 AI automations for service businesses that actually make owners money — Crazy-Movie7633 · 2026-11-17
- SNEWPapers launches remote MCP turning AI tools into US newspaper archive researchers — Jolly-Newspaper-6769 · 2026-11-17
- Team designing open-source mock MCP server, asks community for testing pain points first — OkSkirt1039 · 2026-11-17
- Gemini 4 Hydrogen adds xhigh thinking level, with official guidance on all four tiers — patloeber · 2026-11-17