Sliding-Window Attention Beats Linear Rivals

Episode 1 · Zero-cost sliding-window attention with sinks beats fine-tuned linear attention (2026-08-31, 9 posts)

A new paper by Alexia Jolicoeur-Martineau et al. (with conclusions also published by Microsoft on Hugging Face) reaches a counterintuitive result: on a pretrained softmax Transformer, simply swapping the attention a posteriori for a sliding-window attention (SWA) mask with attention sinks—at zero training cost—outperforms adapting the model to linear attention followed by post-training fine-tuning. If it holds, this challenges the popular route of retrofitting existing models with linear attention to cut inference costs.

Confirmed

• The method requires no post-training; attention sinks are added at zero cost
• SWA matches or beats linear attention models across multiple LLMs and downstream tasks
• On long-context reasoning (Needle-in-a-Haystack, BABILong), SWA outperforms linear attention by 2–10x
• SWA is faster at inference with lower memory footprint; Microsoft calls it a cheaper, more reliable alternative
• Sliding windows restrict each token to a fixed-size neighborhood, sharply reducing attention and KV-Cache memory for long-context models, with periodic global mechanisms补充长程信息

Unconfirmed

• Critics (relayed by ChengleiSi) argue the conclusion is misleading: initializing linear attention from full-attention weights is not the optimal training path, and frontier models typically use hybrid linear/full-attention architectures, so a comparison against the best hybrid remains open

Why it matters

• A nearly free masking change beating expensive post-training adaptation directly challenges the linear-attention retrofit route
• Low memory use makes long-context inference far more viable in memory-constrained settings such as local LLM deployment
• Both attention sinks and sliding windows are mature techniques with low reproduction barriers, likely to be adopted quickly by inference frameworks

• Sliding-Window Attention Beats Linear Attention in Long-Context Tasks — iScienceLuvr · 2026-08-31
• Sliding-window attention beats post-trained linear attention — jm_alexia · 2026-08-31
• New Study: Simple Sliding Window Outperforms Linear Attention — jm_alexia · 2026-08-31
• Sliding-window attention beats quadratic attention in new paper — woadwarrior · 2026-08-31
• Sliding Window Attention Beats Linear Attention Post-Training — jm_alexia · 2026-08-31
• Microsoft: sliding-window attention with sinks beats post-trained linear attention — microsoft · 2026-08-31
• Sliding Window Attention beats linear attention at zero cost in post-training — helloiamleonie · 2026-08-31
• Sliding-window attention beats linear on long-context reasoning — Justgototheeffinmoon · 2026-09-01
• Sliding Window Attention Beats Linear Attention, but Hybrid Architectures Reign Supreme — ChengleiSi · 2026-09-01

Episode 2 · Microsoft paper: sliding window attention beats linear attention in inference (2026-09-02, 2 posts)

A Microsoft Research paper shows that training-free sliding window attention, especially combined with attention sinks, matches or beats post-trained linear attention models and delivers up to 10x better long-context inference while cutting memory.

• Microsoft Paper: Sliding Window Attention Beats Linear Attention for Inference Memory — rohanpaul_ai · 2026-09-02
• Microsoft Research: sliding window attention + sinks beats linear attention 10x on long context — burkov · 2026-09-03