Thom Wolf Releases Free MoE Training Handbook After 800K GPU Hours of Experiments
Thom_Wolf · x · 2026-11-06
Mistral co-founder Thom Wolf notes 18 of the top 20 open models are Mixture-of-Experts — dense is basically over — yet MoE training know-how lives in a handful of teams' heads.
His team spent 800K GPU hours running the experiments and released the free "Ultimate MoE Training Handbook" covering: expert count/size and shared experts, hiding all-to-all latency, mystery 25x slowdowns, router choices (softmax/sigmoid/sqrt(softplus), fp32), aux loss vs DeepSeek's loss-free bias, and upcycling dense models to MoE.
More from Infra
- Vercel launches Schedules public beta: agents can create cron jobs at runtime — cramforce · 2026-11-06
- Harbinger raises $240M Series D as electrification becomes an AI-era bottleneck — NYCounihan · 2026-11-06
- Data center moratoriums are like tariffs: voters want the stance, not the consequences — carlosdponx · 2026-11-06
- Data Centers Shouldn't Build Their Own Power: Off-Grid Gas Turbines Deemed Inefficient and Polluting — aronchick · 2026-11-06
- DeepSeek V4.1 Flash hits 600+ tok/s on Isoquant, 8.7x faster than Claude Opus 5.5 — ycombinator · 2026-11-06
- Iceland pitch to Musk: drill for free geothermal power, build datacenters outside EU reach — djcows · 2026-11-06