focus-llama: a llama.cpp fork implementing Declarative Attention for up to 0.71x decode time
Ok-Shower7286 · reddit · 2026-09-20
A developer implemented Declarative Attention (arXiv:2609.02737, Google DeepMind & KAIST AI) as a llama.cpp fork: the model declares which context chunks it needs via a <focus> tag in its own output, and the engine drops the KV ranges other tokens can no longer attend to — no scorer, no training. Paper-reported vLLM numbers show decode time down to 0.71x (Gemma) and 0.77x (Qwen); the fork itself is not yet benchmarked. Working features include mid-prefill KV range drops, tag-driven one-shot chunk dropping, and a sequence-preserving mode; the post explains why stock llama.cpp couldn't do this (no mid-generation KV access, no paged block table, no kernel-level skipping on CUDA).
More from Infra
- Jevons paradox is classic low-end disruption you won't spot from a GPU-rich hyperlab — cramforce · 2026-09-20
- SpaceX's orbital AI data centers weigh up to 4,000 kg each, filing seeks 1M satellites — XFreeze · 2026-09-20
- Local Models for Personal Agents: GPT Luna Surprises a Coding-Agent Veteran — gized00 · 2026-09-20
- OpenAI hardware VP details first custom chip Jalapeño and its nine-month tape-out — bigdata · 2026-09-20
- VTrain, a Vulkan-based resident trainer, fixes memory leak and offloads more work to GPU — Savantskie1 · 2026-09-20
- vLLM ships day-0 support for Qwen-Image-2.1 with cross-step prefix KV cache — Alibaba_Qwen · 2026-09-20