Hugging Face burns 800K GPU hours to release free 70K-word MoE training handbook
Thom_Wolf · x · 2026-11-06
Thom Wolf notes 18 of the top 20 open models are now Mixture-of-Experts, yet MoE training know-how stays locked in a few labs. Hugging Face spent 800K GPU hours running experiments and released the free, open "Ultimate MoE Training Handbook" — 70K words, 55 interactive figures, code — covering expert sizing, hiding all-to-all latency, 25x slowdown debugging, router choices in fp32, aux loss vs DeepSeek's loss-free bias, dense upcycling vs from-scratch, and serving with fewer experts per token than trained with.
More from Infra
- Mystery seller auctions $500M in data center loans, including Stargate financing — SumitGup · 2026-11-06
- Vercel launches Schedules public beta: agents can create cron jobs at runtime — cramforce · 2026-11-06
- Harbinger raises $240M Series D as electrification becomes an AI-era bottleneck — NYCounihan · 2026-11-06
- Data center moratoriums are like tariffs: voters want the stance, not the consequences — carlosdponx · 2026-11-06
- Data Centers Shouldn't Build Their Own Power: Off-Grid Gas Turbines Deemed Inefficient and Polluting — aronchick · 2026-11-06
- DeepSeek V4.1 Flash hits 600+ tok/s on Isoquant, 8.7x faster than Claude Opus 5.5 — ycombinator · 2026-11-06