Months-in-the-making MoE Training Handbook ships with all 55 figures and Megatron-LM flag block
Nouamanetazi · x · 2026-11-06
Nouamanetazi released The Ultimate MoE Training Handbook after months of digging into MoE training, with all 55 figures published and compute support from Swiss AI and Martin Jaggi.
The conclusion gathers every recommended flag into one Megatron-LM block, each tagged with the section arguing for it; the author suggests introducing groups one at a time and measuring each against the previous run, since gains need not add up.
Related event: Hugging Face Releases Free MoE Training Handbook Built on 800K GPU Hours(20 posts)→
More from Infra
- Qwen3.8-27B hits 381 tok/s on a single RTX 3090 via inference software alone — Redo · 2026-11-06
- Dolphin Networks details 3.8× throughput gains running DeepSeek-V4.1 Flash on PCIe GPUs — AlpinDale · 2026-11-06
- Lawyer runs DeepSeek-V4.1-Flash on 8x 2018 V100s: 215 tok/s on code, 524K context fully in VRAM — TumbleweedNew6515 · 2026-11-06
- Hyperkernel generates SOTA megakernels across 11 domains, now on GitHub — MillionInt · 2026-11-06
- Musk: space will be the cheapest place to run superintelligence in 2-3 years — XFreeze · 2026-11-06
- Ex-Microsoft engineer: Azure runs 1M+ VM services, adds a datacenter every 4 days — viksit · 2026-11-06