Sliding Window Attention Beats Linear Attention Post-Training

jm_alexia · x · 2026-08-31

Research demonstrates that switching a pretrained softmax transformer to a sliding-window attention mask with attention sinks (SWA) outperforms post-training adaptation to linear attention, requiring no additional training. Commentary suggests that for linear attention to work effectively, the architecture should be pretrained with it, rather than adapted later, as invasive changes to strong pretrained models degrade performance.

Related event: Zero-cost sliding-window attention with sinks beats fine-tuned linear attention(9 posts)→

Original post →

More from Research

• Comprehension Audits: New Paper Proposes a Gate for AI-Written Code at Frontier Labs — ronbodkin · 2026-10-16
• Nature paper uses language models to map single neurons behind human language — raphaelmilliere · 2026-10-16
• Reviewer shares workflow: feeding the finished review — never the paper — to AI for a second pass — StingMeleoron · 2026-10-16
• 3SUM record falls to O(n^1.921), Lean-verified — but will it yield usable algorithms? — iskander · 2026-10-16
• Ablation studies to help Kaggle Gemma 4 competition participants — ayushthakur0 · 2026-10-16
• Experts put global AI catastrophe risk before 2100 at 5%, LEAP survey finds — ben_j_todd · 2026-10-16

Research channel →