Language Models Can Control Their Own Attention: 52% Less KV Cache Reading
rohanpaul_ai · x · 2026-09-05
The arXiv paper 'Language Models Can Control Their Own Attention' (Namgyu Ho et al.) introduces Declarative Attention (DA): since models attend to only a small fraction of context yet scan the full KV cache each step, DA has the model declare its attention needs in chain-of-thought, splitting generation into <global> (full context), <focus> (a specific region), and <local> (recent output) modes. The inference engine parses these like tool calls and skips most KV cache reads—no external scorer with O(N) per-step cost.
Zero-shot across 15 long-context tasks:
- Gemma-4-31B: 52.0% fewer attended tokens, only 1.27pp accuracy drop
- Qwen-3.6-27B: 31.1% fewer, 2.75pp drop
Accuracy loss shrinks with model scale, and the authors frame DA as a new axis of sparse attention with further potential under training-based methods.
Related event: Declarative Attention Lets LLMs Cut KV Reads by 52%(3 posts)→
More from Infra
- Dwarkesh publishes 48 flashcards on advanced chip manufacturing to accompany Intel episode — calabi_and_yau · 2026-10-23
- Kazakhstan expands NVIDIA-powered compute as Firebird plans local AI lab — nvidia · 2026-10-23
- Jensen Huang visits Astana, joins Kazakhstan's AI minister to discuss nation's AI future — nvidia · 2026-10-23
- FP16 vs FP32 matmul on RTX 4060: 3.5x speedup at large sizes, slower at 128x128 — cneuralnetwork · 2026-10-23
- BaseQuant V1 quants for agentic Qwen3.8-27B hit 17-34% lower KL than Unsloth — nightowlsleeping · 2026-10-23
- PhD students in some countries get B200s on demand, unlike their US peers — HildeKuehne · 2026-10-23