Shared expert design compared across GLM-5.3, Kimi K3, DeepSeek V4.1 Flash
Nouamanetazi · x · 2026-11-06
The author breaks down shared-expert design in the newest MoEs: GLM-5.3, Kimi K3, and DeepSeek V4.1 Flash all keep a shared expert, while MiMo-V2.6 ships without one; Nemotron 3 Super/Ultra and Kimi K3 also run their experts in a smaller latent space than the model's hidden size. The accompanying figure compares five weighting schemes (routed-only, shared expert, gated shared expert, jointly normalized, parallel dense FFN) by always-on share and tokens dispatched.
More from Research
- Making the planet's surface queryable by AI: Luke Davis unveils months-long project — dansitu · 2026-11-06
- Paul Dirac on where good ideas come from: relaxed Sunday walks, not active striving — CatAstro_Piyush · 2026-11-06
- GPT-6 Astra tops a 16-benchmark robot index, yet went 0/11 on real plug insertions — AIFlow_ML · 2026-11-06
- Probability reasons, statistics quantifies, ML predicts — Tivadar Danka untangles the three — TivadarDanka · 2026-11-06
- Burkov's True Positive Weekly #184: LLMs Can Natively Read & Write Neural Network Weights — burkov · 2026-11-06
- Six Claude agents lived 20 days in a simulated town and spontaneously built tools to cooperate — LiminalWanderings · 2026-11-06