Continuum Attention replaces softmax lookup with continuous integration and wins all paired comparisons
pmttyji · reddit · 2026-10-23
A new paper introduces Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key/value lookup with integration over a continuous piecewise-linear memory curve. Each query induces an exponential relevance density; integrating density and interpolated values yields exact normalized attention via closed-form exponential moments, with gradients from moments of order ≤2 — enabling parallel GPU implementations while keeping quadratic complexity.
CA reduces validation NLL by 0.0200 and 0.0480 nats/byte versus softmax and focal attention respectively, winning all paired comparisons, plus a 0.0457 nats/byte advantage over the strongest softmax baseline on a disjoint slice. Larger-scale generalization remains open. Code and models on GitHub and HuggingFace.
More from Research
- EMNLP oral paper analyzes 16,940 interview responses to define what makes a good qualitative answer — windx0303 · 2026-10-24
- kalomaze shares hilariously incoherent early outputs of tiny byte-level code models — kalomaze · 2026-10-24
- A 17-state Beeping Busy Beaver machine that quasihalts iff the Collatz conjecture is false — rickasaurus · 2026-10-24
- NeurIPS paper OAT: attribute agent failures by learning only from success flows — SharonYixuanLi · 2026-10-24
- Redditor proposes crowdsourced multilingual agent benchmarks to escape English-only evals — Prestigiouspite · 2026-10-24
- Muon optimizer may nearly halve LLM training compute, challenging a decade of AdamW — bycloud · 2026-10-24