Hugging Face burns 800k GPU hours to release the free Ultimate MoE Training Handbook
giffmana · x · 2026-11-06
The Hugging Face team (Thom Wolf, Nouamane Tazi, Lewis Tunstall et al.) released the free, open "Ultimate MoE Training Handbook". Motivation: 18 of the 20 best open models today are Mixture-of-Experts — dense is basically over — yet the know-how to train a good MoE lives mostly in a handful of teams' heads.
To fix that, they spent 800k GPU hours running the experiments themselves and wrote everything down, organized around the key questions training teams face, with polished diagrams and widgets. Sebastian Raschka praised the quality. A rare systematic reference for anyone training MoE models.
Related event: Hugging Face Releases The Ultimate MoE Training Handbook for Free(25 posts)→
More from Models
- Thinking Machines' Inkling 2 Small Docs Openly List the Model's Weaknesses — shizhediao · 2026-11-06
- Small-model customizability called critical as Inkling 2 Small touts SOTA fine-tuning — clarejtbirch · 2026-11-06
- TensorFold 1.2.3 boosts Flash Next decode 21–23%; Apache-2.0 Ripple-65B Preview lands — AIFlow_ML · 2026-11-06
- Researcher alth0u teases that a new model's training run is underway — clarejtbirch · 2026-11-06
- Inkling 2 Small pretrained on 8.4T tokens in 68 hours, natively in NVFP4 — clarejtbirch · 2026-11-06
- Why do Astra and Sol 6.1 perform so well? Multi-agent RL and epistemic vigilance — scaling01 · 2026-11-06