Sliding-window attention beats linear attention, 2-10x on long-context reasoning
s_scardapane · x · 2026-09-14
An arXiv paper by Alexia Jolicoeur-Martineau et al. (Mila) finds that sliding-window attention with sinks matches or beats post-trained linear attention across multiple LLMs and downstream tasks.
- On long-context reasoning (Needle-in-a-Haystack, BABILong), SWA scores 2 to 10 times higher than linear attention
- SWA needs no post-training, is extremely fast and memory-light — a cheap, reliable fix for quadratic attention's memory cost
- Authors strongly recommend switching inference to SWA; linear attention models likely require training from scratch or extensive post-training just to match it
More from Research
- Open-source GeoGuesser RL environment trains VLMs on visual geolocation with GRPO — HuggingEnvs · 2026-09-15
- Jeff Clune discusses a newly possible, powerful type of RL in MIT Tech Review interview — jeffclune · 2026-09-15
- Jeff Clune on a powerful new type of RL in MIT Tech Review interview — jeffclune · 2026-09-15
- Main approaches for finetuning e2e driving models in close(ish)-loop — abursuc · 2026-09-15
- Atria Dawn Preview: student-heavy team launches research-focused agentic base model — xiaohu · 2026-09-15
- Light Origins' humanoid parkour policy picks walk, vault or climb with onboard sensing only — micoolcho · 2026-09-15