Language Models Can Control Their Own Attention: 52% Less KV Cache Reading
rohanpaul_ai · x · 2026-09-05
The arXiv paper 'Language Models Can Control Their Own Attention' (Namgyu Ho et al.) introduces Declarative Attention (DA): since models attend to only a small fraction of context yet scan the full KV cache each step, DA has the model declare its attention needs in chain-of-thought, splitting generation into <global> (full context), <focus> (a specific region), and <local> (recent output) modes. The inference engine parses these like tool calls and skips most KV cache reads—no external scorer with O(N) per-step cost.
Zero-shot across 15 long-context tasks:
- Gemma-4-31B: 52.0% fewer attended tokens, only 1.27pp accuracy drop
- Qwen-3.6-27B: 31.1% fewer, 2.75pp drop
Accuracy loss shrinks with model scale, and the authors frame DA as a new axis of sparse attention with further potential under training-based methods.
Related event: Google paper proposes Declarative Attention, cutting attention cost by 52%(2 posts)→
More from Infra
- Agent outputs die when the VM sleeps: octomind's design for deliverables that survive — donk8r · 2026-09-05
- Japan to develop AI-powered satellites — AIFlow_ML · 2026-09-05
- Hybrid Compute on Mac ships with open-sourced local inference engine and PII classifier — andrewgwils · 2026-09-05
- Tesla's RIM process kills the paint shop, shrinking Cybercab factory footprint ~50% — elonmusk · 2026-09-05
- AMD unveils Threadripper Halo Station: 96-core CPU plus MI350P cards with 288GB HBM3E — Aroochacha · 2026-09-05
- Extropic unveils Z1T, first transformer family for sparse probabilistic hardware with up to 140x GPU energy efficiency — afurgs · 2026-09-05