MoE training paper proposes workload-aware expert-GEMM scheduling
stochasticchasm · x · 2026-07-28
The post highlights a paper excerpt on expert-GEMM scheduling and overlap for MoE training.
- Even when expert load is balanced across ranks, token counts within each rank remain skewed, creating makespan imbalance across SM workers.
- The proposed fix is a workload-aware scheduler that adapts to the current token distribution before launch and keeps parameters fixed during execution.
- A lightweight heuristic uses an analytical cost model of hardware metrics, with coefficients calibrated by offline autotuning.
- For shared experts, GEMMs are dispatched on a separate stream so they can overlap with other kernels.
More from Infra
- SkyPilot says serving Kimi K3 needs multi-node inference and a full stack — skypilot_org · 2026-07-28
- Ilya Sutskever’s SSI partners with Nvidia in a deal Bloomberg pegs at $5 billion — RebeccaBellan · 2026-07-28
- Moonshot’s Mooncake serving stack boosts long-context throughput by up to 525% — stochasticchasm · 2026-07-28
- Fireworks says Kimi K3 matches Opus 5 quality at 2x–4.6x lower task cost — lqiao · 2026-07-28
- Muon orthogonalization goes peer-to-peer to cut all-gather overhead — stochasticchasm · 2026-07-28
- MoonEP keeps MoE training perfectly balanced with dynamic redundant experts — stochasticchasm · 2026-07-28