Declarative Attention lets LLMs declare their own focus, cutting 52% of KV cache reads

eigenlaplace · reddit · 2026-09-05

A new arXiv paper proposes Declarative Attention (DA): instead of pre-selecting tokens with extrinsic proxy scores, the model itself declares where it needs to attend within its chain-of-thought.

Motivation: attention concentrates on a small fraction of context, yet global layers scan the full KV cache each step; proxy-based sparse methods still cost O(N) per step.
Method: generation is partitioned into three modes — <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most KV cache reads.
Results: zero-shot across 15 long-context tasks on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B), DA cuts total attended tokens during decoding by 52.0% and 31.1% with modest accuracy drops (1.27pp / 2.75pp) that shrink with scale.

The authors frame DA as a new axis of sparse attention with further potential under training-based methods.

Original post →

Declarative Attention lets LLMs declare their own focus, cutting 52% of KV cache reads

More from Infra