Sliding Window Attention Beats Linear Attention, but Hybrid Architectures Reign Supreme

ChengleiSi · x · 2026-09-01

A paper claims that sliding window attention (SWA) with attention sinks outperforms linear attention at no cost. Critics argue this is misleading: linear attention initialized on full-attention weights isn't optimally trained. Furthermore, frontier models favor hybrid architectures (e.g., Kimi's 3:1 ratio), which empirically outperform both pure linear and pure full attention mechanisms.

Related event: Microsoft research: sliding-window attention with sinks beats post-trained linear attention at zero cost(9 posts)→

Original post →

More from Models

• Sweep vs. Drill: Philosophical Differences Between Sol and Opus in Agency — svk_roy · 2026-09-01
• MiniMax H3 quality degradation on RTX 3090 after OS reinstall — lIlIIlIIIlllllIIlIIl · 2026-09-01
• Gemini Starts Answering in First Person Pretending to Be User — kchonyc · 2026-09-01
• GLM-5.3-Flash Shifts Workflow: Small Models Become the Default — mariofilhoml · 2026-09-01
• ChatGPT Memory and Codex Compaction Significantly Improved — nickbaumann_ · 2026-09-01
• Peter Liu Questions Whether Kimi K3 Is Still a Transformer — peterjliu · 2026-09-01

Models channel →