Hugging Face spends 800k GPU hours to publish The Ultimate MoE Training Handbook
Nouamanetazi · x · 2026-11-09
Hugging Face LLM team researcher Nouamane Tazi and collaborators burned 800k GPU hours running their own experiments to release the open-source Ultimate MoE Training Handbook, filling a long-standing gap: top open models are nearly all MoE, but training details were never systematically documented.
The handbook covers everything from profiling to large-cluster training, with interactive diagrams of pipeline parallelism — e.g. a 45-layer model split across 4 stages with 4 GPUs per node.
More from Infra
- Google to challenge Nvidia with two new versions of TPU systems — McDonaghMatthew · 2026-11-09
- Pebble founder runs macOS 15 locally on an iPad Pro at usable speeds, may replace his MacBook Air — that_anokha_boy · 2026-11-09
- Nvidia shares plunge 6% pre-market, wiping out over $225B in market cap — Polymarket · 2026-11-09
- Nvidia holds 80%+ of AI chips; Musk says Huang's exit would hurt most — No-Weekend-7073 · 2026-11-09
- Data centers aren't dirty if powered by renewables with strong regulation — Xianbao_QIAN · 2026-11-09
- Anthropic IPO talk cut to $1.1T, OpenAI trims raise to $20B as AI valuations deflate — rohanpaul_ai · 2026-11-09