Hugging Face spent 800k GPU hours writing the ultimate open MoE training handbook
Thom_Wolf · x · 2026-11-06
Hugging Face co-founder Thom Wolf announced "The Ultimate MoE Training Handbook," free and open. Motivation: 18 of the 20 best open models today are MoE, dense is basically over — yet the know-how to train good MoE lives in only a handful of teams.
HF ran 800k GPU hours of experiments and documented everything around the questions training teams actually hit: how many experts and how big, shared or not; why half the step is spent in all-to-all and how to hide it; why a run goes 25x slower with no logged errors; softmax vs sigmoid vs sqrt(softplus) routers and why to keep them in fp32; aux loss vs DeepSeek's loss-free bias; and how to upcycle a dense checkpoint into MoE.
Related event: Hugging Face Releases The Ultimate MoE Training Handbook for Free(25 posts)→
More from Models
- Thinking Machines: Inkling 2 Small live on Tinker, open weights pending safety tests — clarejtbirch · 2026-11-06
- Small-model customizability called critical as Inkling 2 Small touts SOTA fine-tuning — clarejtbirch · 2026-11-06
- TensorFold 1.2.3 boosts Flash Next decode 21–23%; Apache-2.0 Ripple-65B Preview lands — AIFlow_ML · 2026-11-06
- Researcher alth0u teases that a new model's training run is underway — clarejtbirch · 2026-11-06
- Inkling 2 Small pretrained on 8.4T tokens in 68 hours, natively in NVFP4 — clarejtbirch · 2026-11-06
- Why do Astra and Sol 6.1 perform so well? Multi-agent RL and epistemic vigilance — scaling01 · 2026-11-06