Training a MoE Across Compute Islands: 25 Gbps Link, Two Continents, One Model

_arohan_ · x · 2026-11-04

Rohan Anil publishes a position piece and experiment writeup on distributed training across scattered clusters: his team trained a sparse MoE model across two differently-sized, heterogeneous-GPU clusters nearly halfway around the world over a 25 Gbps bidirectional link, and matched standard training by replacing all-reduce with pairwise butterfly-mixing exchanges between eight learners. He argues frontier learning algorithms should be co-designed for a world of fragmented compute, noting the bottleneck is power and grid equipment, not chips: turbines booked through 2028, transformers taking 3+ years, 20+ datacenter projects killed by local opposition in one quarter, and Goldman Sachs expecting only 50-60% of scheduled US datacenter capacity online on time.

Original post →

Training a MoE Across Compute Islands: 25 Gbps Link, Two Continents, One Model

More from Infra