ReCache cuts agentic KV memory 92.43% and speeds first token 3.655x with minimal accuracy loss
techNmak · x · 2026-09-08
arXiv paper 2608.19662 (Yichu Fang et al.) proposes ReCache to fix prefix caching's failure on tool-augmented agents, where schemas recur in varying combinations:
- Resource-wise attention: resource-local positions and no cross-resource attention yield composition-invariant KV blocks
- Structural routing: each resource's KV is exposed only to contribution-selected layer-KV-head groups
- Semantic pruning: keeps only invocation-critical fields (resource name, argument names/descriptions)
On a benchmark from 7 public tool/skill datasets (including resource-disjoint tests), resource-wise attention matches dense invocation performance (82.3% vs 82.4% Inv-F1) with 3.655x time-to-first-token speedup; the full framework cuts allocated KV memory by 92.43% and speeds attention 1.423x. Code is open source.
Related event: ReCache Reuses Tool Schema KV Cache, Cutting VRAM by 92%(2 posts)→
More from coding & agent
- GlossoGen platform systematically studies when LLM agents evolve incomprehensible languages — EliasEskin · 2026-09-08
- W3C × GS1 Zurich meeting pushes two-layer trust framework for agentic commerce — melnykowycz · 2026-09-08
- Claude shipped 583 PRs to a SaaS in one week — and found a payment bypass on its own — mhmazur · 2026-09-08
- PipesHub launches as an open-source permission-aware context layer for AI over company data — Effective-Ad2060 · 2026-09-08
- VibeGame: 8-agent adversarial team turns one sentence into a full playable game project — jiqizhixin · 2026-09-08
- Stageflow: a configurable multi-stage agent pipeline with per-stage sessions and human gates — tejasghutukade · 2026-09-08