Continuum Attention: continuous integration beats softmax attention in all paired comparisons
Afshin Khadangi · hf · 2026-10-15
Afshin Khadangi (University of Luxembourg) proposes Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key/value lookup with integration over a continuous, piecewise-linear memory curve, where each query induces an exponential relevance density. Normalized attention is computed exactly from closed-form exponential moments (including half-token boundary contributions), gradients need moments of order at most two, enabling parallel GPU implementations while keeping quadratic complexity and the standard decoder.
CA cuts mean validation NLL by 0.0200 and 0.0480 nats/byte versus softmax and focal attention respectively, winning all paired comparisons; on a disjoint validation slice it beats the strongest softmax baseline by 0.0457 nats/byte with a paired 95% CI. Larger-scale generalization remains open.
More from Research
- ATAS framework uses world-state memory to keep long-form agentic video generation consistent — zju · 2026-10-15
- Survey from 30+ institutions maps the evolution of multimodal agent memory on two axes — Fudan-University · 2026-10-15
- NJU and JD's Road MOPD tames multi-teacher distillation gradient conflicts, gaining 5+ points — jingdong1 · 2026-10-15
- Stanford and Together AI's HybridAgents: 10.4% gains, up to 3.7x cost savings via latent reasoning — StanfordUniversity · 2026-10-15
- G³VLA injects calibrated multi-camera geometry into VLA models to improve manipulation — Yue Peng · 2026-10-15
- Survey: Towards Agentic Robots organizes the field into 17 categories across four domains — Chonghan Qin · 2026-10-15