EchoMoE Sends Only Deltas Between Nodes, Cutting MoE Inference Traffic 58%

sivareddyg · x · 2026-11-19

EchoMoE tackles the shift of MoE inference from compute-bound to communication-bound: instead of shipping full activations between GPUs hosting different experts, it transmits only the deltas. The retrofitting technique works on pretrained MoEs (no from-scratch training like LatentMoE), cutting inter-node traffic by 58% and boosting throughput by 22.6%.

Original post →

More from Infra

Infra channel →