Hugging Face releases The Ultimate MoE Training Handbook built on 800K GPU hours

Hugging Face author Nouamanetazi and team have published "The Ultimate MoE Training Handbook": practical MoE training experience distilled from roughly 500,000 GPU hours (another account says 800,000 GH200 GPU hours), written up as a 30,000-word blog post with 50+ illustrations (the closing post says 55). Aimed at engineers who have trained dense models and want to move up to MoE, the handbook captures a year of training work on Fable/Astra-generation models. Notably, the author cites Artificial Analysis data showing that 18 of the top 20 open-source models are MoE, evidence that MoE has become the mainstream architecture.

Confirmed

• The handbook walks through how to train MoE with Megatron-LM parameter by parameter, covering the full pipeline from profiling to large-scale cluster training; Chapter 2 is a follow-along runbook: test the hardware, build images, tokenize, train a dense baseline first, then train a first MoE on a single node.
• Routing and load-balancing design comparisons: Kimi K3 uses up to 896 routed experts per layer, drops fixed-step bias updates in favor of quantile-based balancing solved from cross-GPU aggregated score histograms; DeepSeek-V4 uses a fixed hash on token ids to select experts in its first 3 MoE layers, while V4.1-Flash reverts to learnable routing across all layers; DeepSeek-V4 switches router scoring from sigmoid to sqrt(softplus), and on routing scale factors GLM-5.3 and Step 5 Preview stay close to √k while Kimi K3 makes a different choice at k=16.
• Expert structure: GLM-5.3, Kimi K3, and DeepSeek V4.1 Flash keep shared experts; MiMo-V2.6 has no shared experts and pairs with latent compression; Nemotron 3 Super/Ultra and Kimi K3 run experts at smaller dimensions.
• Quantization: Kimi K3 quantizes routed experts to MXFP4 via SFT and RL; DeepSeek-V4 and GLM-5 apply 4-bit QAT after pretraining; NVIDIA trained Nemotron 3 Super (120B total / 12B active params) with NVFP4 on 25T tokens and Nemotron (Ultra) on 20T tokens, keeping critical layers in BF16.
• Optimizer choices: DeepSeek-V4 uses Muon for the router along with the rest of the model; Qwen3.8-Next switched to AdamW after early Muon caused router instability; per-expert Muon has issues with low-frequency expert updates.
• Failure case: in Step 3.5 Flash, the max expert output norm on one layer blew up from about 14x the median to 5700x, completely masked by the training loss; the author simulated this failure plus two other silent failures in videos.
• Inference serving: DeepSeek deploys starting at EP64 (the post gives no further detail).

Why it matters

The handbook is the first systematic side-by-side comparison of design differences across frontier MoE models, complete with a replicable runbook and real failure cases — rare first-hand engineering material for advancing into MoE training. The author thanks lvwerra, Thom Wolf, and others for support and welcomes questions in the discussion thread.

2026-11-06 ~ 2026-11-06 · 19 related posts

• [source] Team spends 500K GPU hours to publish a 30k-word MoE training handbook — Nouamanetazi · 2026-11-06
• 500K GPU-hours MoE Handbook: 18 of Top 20 Open-Weights Models Are MoEs — Nouamanetazi · 2026-11-06
• MoE Training Handbook Ch.2: A Runbook From Hardware Tests to Your First MoE — Nouamanetazi · 2026-11-06
• DeepSeek-V4 switches router scoring to sqrt(softplus); MoE scaling factors diverge — Nouamanetazi · 2026-11-06
• DeepSeek-V4's first layers route by token-id hash, no learned router — Nouamanetazi · 2026-11-06
• Kimi K3 swaps sign updates for quantile balancing across its 896 experts — Nouamanetazi · 2026-11-06
• Shared expert design compared across GLM-5.3, Kimi K3, DeepSeek V4.1 Flash — Nouamanetazi · 2026-11-06
• Shared expert trade-offs: GLM, Kimi, DeepSeek keep one while MiMo ships without — Nouamanetazi · 2026-11-06
• Router optimizer split: DeepSeek keeps Muon while Qwen switched to AdamW after instability — Nouamanetazi · 2026-11-06
• Step 3.5 Flash expert norm blew up 5700x while training loss hid the failure — Nouamanetazi · 2026-11-06
• NVIDIA pretrained 120B and 550B MoEs in NVFP4, keeping last 15% and embeddings in BF16 — Nouamanetazi · 2026-11-06
• MoE handbook: Kimi, DeepSeek, GLM all use 4-bit QAT; DeepSeek EP64 decode 25% faster — Nouamanetazi · 2026-11-06
• [source] Months-in-the-making MoE Training Handbook ships with all 55 figures and Megatron-LM flag block — Nouamanetazi · 2026-11-06
• Author closes out MoE handbook as a record of the Fable/Astra model generation — Nouamanetazi · 2026-11-06
• Hugging Face releases ultimate MoE training handbook from 800k GH200 GPU-hours — lvwerra · 2026-11-06
• [source] Hugging Face burns 800K GPU hours to release free 70K-word MoE training handbook — Thom_Wolf · 2026-11-06

3 near-duplicate retellings: Norlax_42 · Nouamanetazi · Thom_Wolf