Sliding-window attention beats linear attention, 2-10x on long-context reasoning
s_scardapane · x · 2026-09-14
An arXiv paper by Alexia Jolicoeur-Martineau et al. (Mila) finds that sliding-window attention with sinks matches or beats post-trained linear attention across multiple LLMs and downstream tasks.
- On long-context reasoning (Needle-in-a-Haystack, BABILong), SWA scores 2 to 10 times higher than linear attention
- SWA needs no post-training, is extremely fast and memory-light — a cheap, reliable fix for quadratic attention's memory cost
- Authors strongly recommend switching inference to SWA; linear attention models likely require training from scratch or extensive post-training just to match it
More from Research
- ELLIS x UniReps talk: why distinct learning processes yield similar representations — ClementineDomi6 · 2026-10-30
- Physicist calls OpenAI's Navier-Stokes solution physically meaningless despite loophole win — GaryMarcus · 2026-10-30
- Anthropic paper: 0.1% pretraining poison plants RCE backdoor in 4B CLI agent — omarsar0 · 2026-10-30
- CopeNLU presents 7 papers at EMNLP 2026 on bias, alignment, explainability — IAugenstein · 2026-10-30
- Two-stage 3D ResNet pipeline detects neonatal brain pathologies from MRI with 0.919 AUC — Dr_Alex_Crimi · 2026-10-30
- 40 years of putting new neurons into the CNS: why so slow, and can AI accelerate it? — patrickmineault · 2026-10-30