Zero-cost sliding-window attention with sinks beats fine-tuned linear attention

A new paper by Alexia Jolicoeur-Martineau et al. (with conclusions also published by Microsoft on Hugging Face) reaches a counterintuitive result: on a pretrained softmax Transformer, simply swapping the attention a posteriori for a sliding-window attention (SWA) mask with attention sinks—at zero training cost—outperforms adapting the model to linear attention followed by post-training fine-tuning. If it holds, this challenges the popular route of retrofitting existing models with linear attention to cut inference costs.

Confirmed

• The method requires no post-training; attention sinks are added at zero cost
• SWA matches or beats linear attention models across multiple LLMs and downstream tasks
• On long-context reasoning (Needle-in-a-Haystack, BABILong), SWA outperforms linear attention by 2–10x
• SWA is faster at inference with lower memory footprint; Microsoft calls it a cheaper, more reliable alternative
• Sliding windows restrict each token to a fixed-size neighborhood, sharply reducing attention and KV-Cache memory for long-context models, with periodic global mechanisms补充长程信息

Unconfirmed

• Critics (relayed by ChengleiSi) argue the conclusion is misleading: initializing linear attention from full-attention weights is not the optimal training path, and frontier models typically use hybrid linear/full-attention architectures, so a comparison against the best hybrid remains open

Why it matters

• A nearly free masking change beating expensive post-training adaptation directly challenges the linear-attention retrofit route
• Low memory use makes long-context inference far more viable in memory-constrained settings such as local LLM deployment
• Both attention sinks and sliding windows are mature techniques with low reproduction barriers, likely to be adopted quickly by inference frameworks

2026-08-31 ~ 2026-09-01 · 9 related posts

Full story(2 episodes)→

• Episode 1: Zero-cost sliding-window attention with sinks beats fine-tuned linear attention(2026-08-31, 9 posts)
• Episode 2: Microsoft paper: sliding window attention beats linear attention in inference(2026-09-02, 2 posts)

• Sliding-Window Attention Beats Linear Attention in Long-Context Tasks — iScienceLuvr · 2026-08-31
• New Study: Simple Sliding Window Outperforms Linear Attention — jm_alexia · 2026-08-31
• Sliding-window attention beats quadratic attention in new paper — woadwarrior · 2026-08-31
• Sliding Window Attention Beats Linear Attention Post-Training — jm_alexia · 2026-08-31
• [source] Microsoft: sliding-window attention with sinks beats post-trained linear attention — microsoft · 2026-08-31
• Sliding Window Attention beats linear attention at zero cost in post-training — helloiamleonie · 2026-08-31
• Sliding Window Attention Beats Linear Attention, but Hybrid Architectures Reign Supreme — ChengleiSi · 2026-09-01

2 near-duplicate retellings: jm_alexia · Justgototheeffinmoon