Microsoft Research: sliding window attention + sinks beats linear attention 10x on long context
burkov · x · 2026-09-03
A new Microsoft Research paper shows simple sliding window attention with attention sinks matches or exceeds post-trained linear attention models on standard benchmarks, and outperforms them by up to 10x on long-context reasoning — with no post-training required. The paper is now readable with an AI tutor on ChapterPal, despite its restrictive license.
More from Research
- O(n) algorithm found for optimal multinomial Kelly wagering, beating naive O(n log n) — alexbilz · 2026-10-16
- Beam LoRA fine-tuning benchmark: 500 steps, 16M tokens, 16 minutes on one 8x B300 node — QGallouedec · 2026-10-16
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16