Four Scheduling Techniques Flatten MoE Training Memory Peaks, Enabling 1M Context at 10.4x Throughput
Shrey Pandit · hf · 2026-09-17
MoE training at long context or large batch fails the moment any single component's peak memory exceeds device capacity. The paper identifies four unbounded peaks in common parallelism plans: expert dispatch (routing matrix), vocabulary projection (tokens × vocab), gradient checkpoint boundaries (depth × sequence length), and optimizer states.
Four schedules bound all peaks by changing only the order and granularity of computation and data movement, keeping loss and gradients exact:
- PipelinedLLEP: least-loaded expert parallelism with per-source dispatch caps — dispatch peak cut up to 59.3% without throughput loss
- Ring-DTP: ring circulation of activations/weight shards plus online log-sum-exp folding of logit blocks — vocab projection peak cut 86.6%
- SCO: offloads each checkpoint boundary's long-lived tensor to CPU
- OffloadStreamAdamW: pipelines the serial CPU Adam update — offloaded optimizer steps 2.05x faster
Composed on MoE models from 120B to 667B, they train at 1M context — 8–32x the reach of a tuned FSDP2 baseline and up to 10.4x its throughput.
More from Infra
- Speculative decoding: small draft model proposes tokens, big model verifies in one pass — HowDevelop · 2026-09-17
- India to invest roughly $30 billion in building a domestic semiconductor industry — Polymarket · 2026-09-17
- Dev Inspired by AgentConf Keynote to Run All Agents Locally on Home Blackwell GPU Stack — TejasKumar_ · 2026-09-17
- DeepSeek-V4.1 Flash deep dive: pushing KV cache compression to the limit at 420 tok/s — teortaxesTex · 2026-09-17
- GLM agent built its own inference infra in two weeks, tripling end-to-end throughput — jietang · 2026-09-17
- Huawei Unveils Ascend 960 Chips and SuperPoD Stack Aiming at Million-Card Clusters — 智东西 · 2026-09-17