Trained an MoE across two GPU clusters 25 Gbps apart — the compute islands guide

_arohan_ · x · 2026-11-05

Rohan Anil published "The Hitchhiker's Guide to Training on Compute Islands," a position paper and call to action on truly distributed learning, with a working proof: training a sparse MoE across two differently-sized clusters with different GPU types, nearly halfway around the world, over a 25 Gbps bidirectional link. Revisiting butterfly mixing with pairwise exchanges between eight learners matched standard training.

Key points:

Training compute rarely arrives as one contiguous block; scattered clusters of hundreds to thousands of GPUs across neoclouds add up substantially
The bottleneck isn't chips but power, grid equipment, and land — Satya Nadella: "you may actually have a bunch of chips sitting in inventory that I can't plug in"
Gas turbines booked through 2028 (Bloomberg); large transformers take 3+ years (Reuters); local opposition killed 20+ projects in a single quarter (Heatmap)
Goldman Sachs expects only 50-60% of scheduled US data center capacity online on time over the next 1-2 years, down from 72% historically

Original post →

Trained an MoE across two GPU clusters 25 Gbps apart — the compute islands guide

More from Infra