focus-llama: a llama.cpp fork implementing Declarative Attention for up to 0.71x decode time
Ok-Shower7286 · reddit · 2026-09-20
A developer implemented Declarative Attention (arXiv:2609.02737, Google DeepMind & KAIST AI) as a llama.cpp fork: the model declares which context chunks it needs via a <focus> tag in its own output, and the engine drops the KV ranges other tokens can no longer attend to — no scorer, no training. Paper-reported vLLM numbers show decode time down to 0.71x (Gemma) and 0.77x (Qwen); the fork itself is not yet benchmarked. Working features include mid-prefill KV range drops, tag-driven one-shot chunk dropping, and a sequence-preserving mode; the post explains why stock llama.cpp couldn't do this (no mid-generation KV access, no paged block table, no kernel-level skipping on CUDA).
More from Infra
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12
- Megakernel is multiprogramming for GPUs: 'GPUs are entering the CPU age' — pranjalssh · 2026-11-12