Hugging Face spent 800k GPU hours to publish a free Ultimate MoE Training Handbook
gordic_aleksa · x · 2026-11-07
Thom Wolf notes 18 of the 20 best open models today are Mixture-of-Experts — "dense is basically over" — yet MoE training know-how lives in a handful of teams' heads. So the team spent 800k GPU hours running experiments themselves and released "The Ultimate MoE Training Handbook", free and open, organized around the key questions training teams face.
Related event: Hugging Face Releases Free MoE Training Handbook After 800K GPU-Hours(27 posts)→
More from Research
- GLASS preprint teaser: graph-led LLM-assisted scoring of open-ended responses without human coding — GolinoHudson · 2026-11-07
- LLMs can read and patch weights, but 20 fine-tuning steps erase the edits — kevinpl07 · 2026-11-07
- FlexiTac-Gen launches: AI-agent-assisted tool to design custom tactile sensors, accepted to CoRL 2026 — Haoyu_Xiong_ · 2026-11-07
- PolicyIK: NYU trains RL for inverse kinematics on a real arm, hitting 0.5mm median error — Lianhuiq · 2026-11-07
- Weight-space programming vs LoRA on 4 bugs: precise one-neuron patches that don't persist — kevinpl07 · 2026-11-07
- Microsoft Research's Code Video Model generates video from editable code, scoring 85.82 vs VACE's 80.59 — jiqizhixin · 2026-11-07