Sliding-window beats linear attention
Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, Emy Gervais
cs.CL, cs.LG
2026-08-28
Sliding-window attention with 4 sinks and no post-training recovers 99.0% of a 6-benchmark average across 11 LLMs and beats linearized models 2 to 10x on long-context tasks.
Retrofitting a pretrained LLM with linear attention is supposed to freeze inference memory: a recurrent state instead of a KV cache that grows forever. LoLCATs advertised a 40M-token conversion; QRWKV spent a few hundred million tokens turning Qwen into an RWKV variant. Those papers compared against sliding-window attention without sinks. Once the first tokens slide out of the window, that baseline collapses, because transformers park leftover attention on the opening positions.
The comparison that was missing is sliding-window attention that always keeps the first four tokens. This paper runs that mask change, with no weight updates, against a long list of post-trained linearizers.
At inference, replace the causal mask with SWA(w, 4): each token attends to the last w positions plus four sinks at the start of the sequence. Default w is 64. Weights stay frozen. Stacked layers grow the effective receptive field to roughly layers times window, so short-form benchmarks are not actually limited to 64 tokens of context.
They score 11 published linearization recipes, including SUPRA, Hedgehog, LoLCATs, Liger-GLA, MOHAWK, Mamba-in-the-Llama, DiJiang, ARWKV, Llamba, QLinAtt, and QRWKV6/7, on bases from Phi-1.5 1.3B through Llama-3.1-70B and Qwen2.5-72B. Short tasks: 5-shot MMLU plus ARC-C/E, HellaSwag, PIQA, Winogrande. Long tasks: S-NIAH and BABILong on Llama-3.1-8B, windows 128/256/512, lengths up to 4K. Speed and memory use a 4-layer, 1024-d transformer, FlashAttention for softmax/SWA and ThunderKittens for linear kernels.
They also distilled Qwen3-8B, Phi-4-mini-reasoning, and Phi-4-reasoning-plus themselves, LoLCATs-style, on about 0.1B tokens of cleaned Alpaca, into GLA, Gated DeltaNet, and QRWKV6. That avoids relying only on numbers from other papers.
SWA(64,4) wins the six-benchmark average in 9 of 11 settings. Across models it recovers 93.2% of teacher MMLU and 99.0% of the six-task average. QRWKV6 is 92.4% / 99.1% after 350 to 700M tokens. LoLCATs recovers 83.2% of MMLU from 40M tokens.
Two near-ties: on Phi-1.5, LoLCATs averages 62.5 versus SWA 62.4; on Qwen2.5-32B-Instruct, QRWKV6 averages 77.3 versus SWA 76.6 (teacher 77.2).
| Setup | MMLU recovery | 6-task average recovery | Post-training tokens |
| SWA(64,4) | 93.2% | 99.0% | 0 |
| QRWKV6 | 92.4% | 99.1% | 350–700M |
| LoLCATs | 83.2% | 97.5% | 40M |
| Liger-GLA | 62.2% | 92.0% | 20M |
Per-model: Llama-3.1-8B teacher 72.5 average, SWA 71.8, LoLCATs 70.3. The 70B teacher is 79.1, SWA 78.2, LoLCATs 75.6. The drop is almost all MMLU; commonsense items barely move.
Long context is the split. On Llama-3.1-8B, window 256, 4K: S-NIAH-3 scores 19.6 (SWA), 2.2 (LoLCATs), 0.6 (Liger-GLA), 99.8 (full attention). BABILong averages 15, 3, and 60. That matches the "2 to 10 times" claim. Relative to full attention at 4K, SWA keeps 17.2% to 23%; the linearizers stay in the single digits.
The in-house modern-architecture distillations look worse: Qwen3-8B teacher 72.5, SWA 71.6, Gated DeltaNet 56.1.
SWA is the flattest and fastest decoder; full attention slows after 1K. Window-64 SWA uses the least memory; pure linear is next; window-512 SWA sits above pure linear. If memory is the constraint, shrink the window rather than swap the kernel.
If the goal is bounded inference memory without touching weights, changing the mask is enough. Linearization papers used a crippled SWA baseline. With sinks restored, post-trained linearizers at best match on short tasks and lose on long ones.
Practical order of operations: ship SWA(64 or 256, 4) first, then decide whether tens of millions of tokens of linearization are worth it. Winning with linear attention likely means training from scratch; converting a softmax model is not enough.
A related datapoint on video: Sliding Tile Attention recovers 97% of HunyuanVideo's VBench score with no training, at 3.53 times the speed.
The authors only test training-free SWA; extra post-training (SWAA) might lift it further. No hybrid stacks with a few full-attention layers. No agentic evals, no truly large serving study, and multimodal is mentioned rather than run.
A few claims overreach. Long context stops at 4K, where full attention still near-perfects S-NIAH and SWA keeps about a fifth of that. That is "collapses less than linearization," not a long-context solution. Speed numbers come from a 4-layer toy model and should not be read as 70B serving. LoLCATs already mixes in SWA, so pure SWA versus linear-plus-SWA tying on short tasks says the linear branch added little under that budget. The paper does not explain why MMLU drops more than commonsense.