Sliding-window attention beats linear on long-context reasoning
Justgototheeffinmoon · reddit · 2026-09-01
A new arXiv preprint claims that Sliding Window Attention (SWA) with sinks outperforms linear attention variants—which require significant post-training compute—on long-context reasoning tasks.
Key Findings
- On benchmarks like Needle-in-a-Haystack and BABILong, SWA achieves 2 to 10 times higher performance than linear attention.
- Existing linear attention research may have benchmarked against the wrong baselines, lacking proper comparison to simple methods like SWA.
Advantages & Recommendation
- SWA requires no post-training, offers fast inference, and maintains low memory usage.
- The authors strongly recommend switching to SWA over post-trained linear models, noting that linear attention likely requires training from scratch or extensive post-training just to match SWA.
More from Research
- Xiaomi's CodeMidas turns source code into RL environments: 5,545 tasks, 23 languages — teortaxesTex · 2026-09-21
- Safety follows understanding: Nina Miolane calls for predictive laws of intelligence — ninamiolane · 2026-09-21
- Nina Miolane argues AI safety needs 'engineering laws': predictive math for intelligence — ninamiolane · 2026-09-21
- UC Berkeley launches Science of Intelligence Institute; Nina Miolane to give inaugural lecture Oct 7 — ninamiolane · 2026-09-21
- Tim Dettmers' lab announces open-source week: 2 frameworks, 4 papers for frontier AI on local hardware — Tim_Dettmers · 2026-09-21
- Radial Duality: turning constrained optimization into unconstrained Lipschitz problems — prof_grimmer · 2026-09-21