DeepSeek-V4 switches router scoring to sqrt(softplus); MoE scaling factors diverge
Nouamanetazi · x · 2026-11-06
How should router scores become weights? DeepSeek-V4 switched from sigmoid to sqrt(softplus). On the routed scaling factor the newest MoEs split: GLM-5.3 and Step 5 Preview stay near √k, while Kimi K3 ships 1.0 at k=16 and normalizes the routed sum before its up-projection. Practical tip: bf16 can round two close logits together and flip which experts win, so always set --moe-router-dtype fp32. The figure compares four normalization schemes, showing only √k keeps output RMS near a single expert's on every token.
More from Research
- Making the planet's surface queryable by AI: Luke Davis unveils months-long project — dansitu · 2026-11-06
- Paul Dirac on where good ideas come from: relaxed Sunday walks, not active striving — CatAstro_Piyush · 2026-11-06
- GPT-6 Astra tops a 16-benchmark robot index, yet went 0/11 on real plug insertions — AIFlow_ML · 2026-11-06
- Probability reasons, statistics quantifies, ML predicts — Tivadar Danka untangles the three — TivadarDanka · 2026-11-06
- Burkov's True Positive Weekly #184: LLMs Can Natively Read & Write Neural Network Weights — burkov · 2026-11-06
- Six Claude agents lived 20 days in a simulated town and spontaneously built tools to cooperate — LiminalWanderings · 2026-11-06