Declarative Attention lets LLMs cut KV cache scanning by up to 52% with minimal accuracy loss
burny_tech · x · 2026-09-04
The problem
Long-context LLMs waste compute by scanning the entire KV cache at every decoding step, even when only a small region matters.
The approach
The paper introduces Declarative Attention, where the model explicitly switches between global, focused, and local attention during reasoning, letting the inference engine skip irrelevant KV cache regions.
Results
- Gemma-4-31B: 52% fewer attended tokens, only 1.27pp accuracy drop
- Qwen-3.6-27B: 31% fewer tokens, 2.75pp drop
A lightweight mechanism that trades tiny accuracy for major inference efficiency gains in long-context reasoning.
Related event: Declarative Attention Lets LLMs Control Their Own KV Cache Reading(5 posts)→
More from Research
- Seqlet caller follow-up: credits for DeepLIFT contributors — jmschreiber91 · 2026-10-21
- Center for AI Safety Publishes Full Syllabus for Intro to ML Safety Course — ArtificialOther · 2026-10-21
- Seqlet caller gets 22x speedup: 167,750 loci drop from 7.8s to 0.35s — jmschreiber91 · 2026-10-21
- X trends: SophontAI's Holon-S1 predicts gene activity from cancer slides; decision models rise — iScienceLuvr · 2026-10-21
- NEJM AI study: LLM-recommended care costs up to 10x more than physician orders — dvir_a · 2026-10-21
- Nucleotide dependency analysis turns genomic language models into functional element detectors — jmschreiber91 · 2026-10-21