SWA vs linear attention: hybrid beats sliding window 1.8x on long math answers
HochreiterSepp · x · 2026-11-03
Researchers distilled Qwen3-4B into two variants under identical KV-cache budgets and 4.1B training tokens: pure SWA vs a SWA+mLSTM hybrid (Bifocal). On short answers they tie (59.1% vs 59.6% pass@1), but on long math answers (7.6K tokens; the 96 problems with answers over 4K), the hybrid solves 1.8× as many (4.4% vs 2.5% pass@1).
Takeaway: sliding window attention is fine for short outputs but drops critical context on long-form generation, and mixing in linear attention (mLSTM) closes the gap substantially — a quantified answer to whether SWA really beats linear attention.
More from Models
- GPT-6 Astra edits 0.016% of Qwen3.5-0.8B weights, boosting math up to 6x with no training — _lewtun · 2026-11-03
- Alignment debate: treat AI as allies—slow down and stop deceiving them — davidmanheim · 2026-11-03
- Alleged 14T Anthropic "numbat" leak debunked: weights regenerate from name-seeded random noise — djcroman · 2026-11-03
- Altman: Next GPT Solved Navier-Stokes With 10,000 Agents Working 4 Days Straight — DeryaTR_ · 2026-11-03
- DeepSeek V5 reportedly planned to be pre-trained on Huawei Ascend chips — hedgehog0 · 2026-11-03
- Loyal Google user: Gemini replaced Assistant and everything got worse — alpinesteveo · 2026-11-03