Sliding Window Attention Beats Linear Attention Post-Training
jm_alexia · x · 2026-08-31
Research demonstrates that switching a pretrained softmax transformer to a sliding-window attention mask with attention sinks (SWA) outperforms post-training adaptation to linear attention, requiring no additional training. Commentary suggests that for linear attention to work effectively, the architecture should be pretrained with it, rather than adapted later, as invasive changes to strong pretrained models degrade performance.
More from Research
- O(n) algorithm found for optimal multinomial Kelly wagering, beating naive O(n log n) — alexbilz · 2026-10-16
- Beam LoRA fine-tuning benchmark: 500 steps, 16M tokens, 16 minutes on one 8x B300 node — QGallouedec · 2026-10-16
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16