Hugging Face team spent $2M training MoEs — and released it all as a public handbook
Nouamanetazi · x · 2026-11-06
The Hugging Face team (Nouamanetazi, lvwerra, Thom Wolf) released a handbook on training Mixture-of-Experts LLMs, saying they spent $2M to learn these lessons.
MoEs power the AI models people use daily, yet very few people worldwide know how to train them at scale — only a handful at each top lab. The handbook distills everything into carefully crafted chapters and figures, may take days to read, and aims to enable 1000x more people to train their own frontier models.
Related event: Hugging Face Releases Ultimate MoE Training Handbook After 800K GPU Hours(21 posts)→
More from Research
- Meta's Axia Learns Scaling Laws for Ranking Models from a Few Trial Runs — _reachsumit · 2026-11-06
- Meta's CAP-Mix Replaces Hard Semantic IDs with Soft Prototype Mixtures for Generative Retrieval — _reachsumit · 2026-11-06
- Stanford team proposes Joules-to-GDP: measuring AI's economic value per joule of energy — niloofar_mire · 2026-11-06
- Weaver: Incremental Reasoning Over Conversation Memory for Personalized RAG — _reachsumit · 2026-11-06
- METR report on AI accelerating AI R&D flips a skeptic: >50% on superintelligence in 18 months — GaryMarcus · 2026-11-06
- Optimal Re-Ranking Depth Varies by Query: LLM-Judged Predictor Cuts Depth 3x — _reachsumit · 2026-11-06