Microsoft Paper: Sliding Window Attention Beats Linear Attention for Inference Memory
rohanpaul_ai · x · 2026-09-02
A new Microsoft paper finds that for reducing inference memory, training-free Sliding Window Attention (SWA) outperforms most retrofitted linear-attention methods.
Methodology: Keep only a small recent window plus the first 4 “sink” tokens that models rely on.
Key Findings:
- With a 64-token window, this setup achieved the best average downstream score in 9 of 11 model comparisons and recovered 99.0% of the full-attention baseline.
- The performance gap widened on long-context reasoning: at 4K context, SWA reached 17.2%–23.0% on Needle-in-a-Haystack tasks (vs. 5.8% for LoLCATs) and 15% on BABILong (vs. 3%).
- In speed and memory tests, the 64-token SWA setup was the fastest and most memory-efficient.
Conclusion: While full attention still wins on very long contexts, SWA with attention sinks is recommended as the first try for fixed, low-memory scenarios without retraining.
More from Research
- Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
- semiconductor.diy: open-source guides to make real chips in a garden shed — jwt0625 · 2026-10-18
- Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
- Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs — inductionheads · 2026-10-18
- q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18
- Smarter Every Day's Destin to defend PhD dissertation on impact fusion — CatAstro_Piyush · 2026-10-18