Chinese Frontier Models Shift to Linear Attention and Open Source Strategy
SumitGup · x · 2026-08-27
Elie Bakouch notes that almost all Chinese frontier models, except DeepSeek, have adopted linear attention mechanisms. Most, except Kimi, also utilize sparse attention with similar indexers to maximize compute and memory efficiency. They widely use advanced residuals (mHC, attention residual) and the Muon optimizer. Kyle Li adds that this collaborative open-source approach reflects a belief that the whole is greater than the sum of parts, driving an exciting era for efficient OSS models.
Related event: Chinese Frontier Models Shift to Linear Attention and Open Source(2 posts)→
More from Models
- Google says teams across the company are using Gemini 4 Argon on its hardest engineering problems — CurieuxExplorer · 2026-10-15
- Perplexity's new embedding models: index with a 9B model, search with 0.6B — antoine_chaffin · 2026-10-15
- 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
- Leak: Moonshot's Kimi flagship 'x4' done with pre-training, K3.1 in internal testing — lyraxana · 2026-10-15
- Unsloth ships GGUF quants for Nemotron 3.5: Bolt-74B runs in 24GB RAM — danielhanchen · 2026-10-15
- NVIDIA ships two open Nemotron 3.5 Bolt models: a 74B-A9B MoE and a mobile-friendly E4B — HankYeomans · 2026-10-15