Declarative Attention Cuts LLM Attention Work by 52%
rohanpaul_ai · x · 2026-09-05
A Google deployment paper introduces Declarative Attention: instead of rereading the whole context for every token, the model itself declares which context regions it needs and the inference engine skips the rest—no external scorer required. Zero-shot on 15 long-context tasks, attended tokens fell 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B, with accuracy drops of just 1.27pp and 2.75pp. Larger models handle the trade-off better, suggesting further gains with training.
Related event: Declarative Attention Lets LLMs Cut KV Reads by 52%(3 posts)→
More from Infra
- Jensen Huang visits Astana, joins Kazakhstan's AI minister to discuss nation's AI future — nvidia · 2026-10-23
- FP16 vs FP32 matmul on RTX 4060: 3.5x speedup at large sizes, slower at 128x128 — cneuralnetwork · 2026-10-23
- BaseQuant V1 quants for agentic Qwen3.8-27B hit 17-34% lower KL than Unsloth — nightowlsleeping · 2026-10-23
- PhD students in some countries get B200s on demand, unlike their US peers — HildeKuehne · 2026-10-23
- Nvidia's Berlin GTC spotlights humanoid robots and Vera Rubin AI supercomputing — emmanuelvivier · 2026-10-23
- Lenovo adds 5 AI Library use cases, claims 1-week deployment and up to 70% lower token cost — shashib · 2026-10-23