New Study: Simple Sliding Window Outperforms Linear Attention
jm_alexia · x · 2026-08-31
A new study presents a counter-intuitive finding: a simple sliding-window attention mask with attention sinks (added at no cost) outperforms linear attention that has undergone post-training. The paper notes this approach requires no extra post-training, reduces memory usage, and achieves better scores.
More from Research
- O(n) algorithm found for optimal multinomial Kelly wagering, beating naive O(n log n) — alexbilz · 2026-10-16
- Beam LoRA fine-tuning benchmark: 500 steps, 16M tokens, 16 minutes on one 8x B300 node — QGallouedec · 2026-10-16
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16