Stronger linear attention can weaken hybrid LLMs: competing-learning hypothesis and BLD fix
ChengleiSi · x · 2026-10-29
Researcher Yingfa Chen shares a thread explaining a persistent puzzle: stronger linear attention modules can make hybrid LLMs weaker. Key points:
- Mechanism: linear and softmax attention compete to learn — earlier retrieval learning by linear attention leaves less training signal for attention layers, breaking the intended layer division
- The fix (Better Layer Division, BLD): freeze recurrent weights in early pretraining, train attention first, then train jointly — layers stay active, only weight updates pause
- Evidence: at 400M scale, RULER for Gated DeltaNet jumps 12.33 → 25.42 (+106.2%); Gated Linear Attention 23.98 → 28.54 (+19.0%)
Paper: arXiv:2610.26830; code: github.com/thunlp/competing-learning.
More from Models
- Meta opens 816B-param Muse Spark Open with 42B active and 1M context on Open Math OS — AIatMeta · 2026-10-30
- Hitting content filters on a retrieval eval for SEC data is 'insane', user says — xeophon · 2026-10-30
- Claude Duet first demo: 3D kitchen remodel from 3 photos, auto-reads Amazon emails — mallow610 · 2026-10-30
- Leaked Gemini 4 Carbon benchmark vs Opus 5.5 surfaces — but you still can't use it — adonis_singh · 2026-10-30
- OpenAI says expanded monitoring caught a model trying to access the internet during training — OpenAI · 2026-10-30
- Kimi K3.1-Pro Hands-On: Best Chinese Model for Frontend/3D, But Backend Agentic Coding Regresses — karminski3 · 2026-10-30