Hugging Face releases MoE training handbook distilled from 800k GH200 GPU-hours
Norlax_42 · reddit · 2026-11-06
Hugging Face published "The Ultimate MoE Training Handbook" for engineers moving from dense to Mixture-of-Experts training:
- Chapter 2 is a Megatron-LM runbook: test hardware, build the image, train a dense baseline, then a first MoE on one node.
- Chapter 3 dissects the MoE layer (router, routing, load balancing, experts, block layout, systems) with Megatron-LM code, recommendations, and what frontier MoEs chose; Chapter 4 covers serving and RL. Chapters 3–4 need no GPU.
- Lessons come from 800k GH200 GPU-hours on the CSCS Alps cluster; own runs, paper citations, and toy simulations are clearly labeled.
Released alongside: a 55-figure gallery, scripts, and one Megatron-LM block with every recommended flag.
Related event: Hugging Face Releases Free MoE Training Handbook Built on 800K GPU Hours(20 posts)→
More from Infra
- Qwen3.8-27B hits 381 tok/s on a single RTX 3090 via inference software alone — Redo · 2026-11-06
- Dolphin Networks details 3.8× throughput gains running DeepSeek-V4.1 Flash on PCIe GPUs — AlpinDale · 2026-11-06
- Lawyer runs DeepSeek-V4.1-Flash on 8x 2018 V100s: 215 tok/s on code, 524K context fully in VRAM — TumbleweedNew6515 · 2026-11-06
- Hyperkernel generates SOTA megakernels across 11 domains, now on GitHub — MillionInt · 2026-11-06
- Musk: space will be the cheapest place to run superintelligence in 2-3 years — XFreeze · 2026-11-06
- Ex-Microsoft engineer: Azure runs 1M+ VM services, adds a datacenter every 4 days — viksit · 2026-11-06