Meta Open-Sources Dist-MoE, Speeding MoE Training Up to 1.40x on GB300 NVL72
ezyang · x · 2026-10-21
Meta SuperIntelligence Labs open-sourced Dist-MoE, a speed-of-light distributed Mixture-of-Experts kernel library. PyTorch's ezyang says it has essentially solved cudagraphable, dropless MoE both internally and in OSS. It features fused communication-compute kernels for BF16/MXFP8 training and NVFP4 inference, full CUDA graph support, dynamic activation packing into bounded buffers, and capacity-driven recompute — no shared-expert branch needed. On GB300 NVL72, it hit up to 1.40x speedup over Mixture-of-Kittens (MoK) on an imbalanced ten-layer MXFP8 training workload across 64 GPUs. Ships with an illustrated interactive guide.
More from Infra
- xAI said to lean on ACC/AEC and copper as Chinese pluggables stay off-limits — jwt0625 · 2026-10-21
- Packed a16z SF infra event features fal, ElevenLabs, BFL and Krea engineers — venturetwins · 2026-10-21
- Baseten ships DeepSeek v4.1 Flash, Reflection Beam 1.0, Nemotron Super 3.5 VL day-zero with claimed top benchmarks — baseten · 2026-10-21
- Modal Engineers Detail Sticky Serverless Routing for Real-Time AI Inference — blelbach · 2026-10-21
- EdgeKernelBench: LLM-Generated Kernels Hit 64.5% Correct but Only 25.5% Fast on Edge Backends — encodelab · 2026-10-21
- PyTorch accelerator integration group details progress and priorities at PyTorchCon — PyTorch · 2026-10-21