Kimi K3 uses normalized latentMoE for training stability, norm details revealed
eliebakouch · x · 2026-10-16
HF researcher Elie Bakouch notes that Kimi K3 employs a normalized "latentMoE" to stabilize training, sharing the full norm list from the Muse codebase: 9 normalization sites from inputlayernorm, qknorm, and onorm to pre/postfeedforwardlayernorm, 5 of them learnable (normalize-then-scale by 1+w, or scale-then-normalize), with a single postexpertnorm module shared by all 256 experts and widths alternating between 8192 and 4096.
More from Models
- EpochAI pegs GPT-6 training at up to 1.8e27 FLOP; scaling01 calls param estimates way too high — scaling01 · 2026-10-16
- Gemini 4 Argon reportedly now available to all Antigravity Pro users — clmt · 2026-10-16
- Mythos on Claude's 'smallness': accurate proprioception, not low self-esteem — repligate · 2026-10-16
- Meta's Muse Spark Open scores 39 vs 48 for closed Spark 1.3 on AA Intelligence Index — EAccelerate_42 · 2026-10-16
- PhoneLLM Mini (30B) and Max (120B) target sub-100ms voice AI latency on Workers AI — ritakozlov · 2026-10-16
- Yacine on LLMs: "I'm sick of how stupid these language models are. Accelerate!" — yacineMTB · 2026-10-16