Sliding-window attention beats post-trained linear attention
jm_alexia · x · 2026-08-31
A new paper shows that Sliding Window Attention (SWA) with sinks matches or outperforms post-trained Linear Attention models at no cost. SWA achieves 2-10x higher performance on long-context reasoning tasks. It requires no post-training, is fast, and low-memory. The authors recommend switching to SWA over retrofitting linear attention to reduce inference memory costs.
More from Research
- A week of ChatGPT Pro quota burned in 36 hours as GPT-6 fails to factor a 512-bit RSA key — matthew_d_green · 2026-10-16
- Kambhampati on how System 1 LLMs acquire System 2 competencies — open access in AI Magazine — rao2z · 2026-10-16
- Qdrant tests Perplexity's visual PDF embeddings: 21x smaller vectors, 99.9% after rescoring — jrxna · 2026-10-16
- Tessellation-free geometric queries on displaced surfaces cut caustics variance 10x — ssh4net · 2026-10-16
- MIT's Buehler: AI as 'a new language of science,' three years on — ProfBuehlerMIT · 2026-10-16
- Mere availability of AI advice collapses human "I don't know" from 43% to 3% — ValerioCapraro · 2026-10-16