MoE All-to-All Comms Eat 60% of Training Steps; Correlated Placement Cuts 58% of Dispatched Rows

QuentinAnthon15 · x · 2026-11-06

Essential AI researcher Quentin Anthony and co-authors released an arXiv paper on reducing all-to-all communication overhead in MoE pretraining under expert parallelism.

Key findings:

On a cluster with 8 AMD Instinct MI300X GPUs per node, all-to-all collectives take 45% of the training step at EP32 with top-2 routing, and 60% with top-6 routing.
Early in pretraining, routers already learn correlated assignment patterns: at top-2, 0.8% of expert pairs in a layer are jointly selected by 42% of tokens, and a token's expert choices at one layer predict its choices at the next.

Optimizations:

Correlated expert placement: co-locating frequently co-selected experts on the same GPU;
Combined with a dispatcher that sends each token to each GPU only once, it removes up to 58% of dispatched rows;
Token shuffling: under sequence parallelism, moving each token to the GPU hosting its experts to cut cross-GPU/node traffic.

The author notes these complexities emerged from testing new attention kernels on real models and real input distributions, revealing surprising overlap between interpretability work and systems research.

Original post →

MoE All-to-All Comms Eat 60% of Training Steps; Correlated Placement Cuts 58% of Dispatched Rows

More from Infra