Training an MoE across continents on 25 Gbps: Core Automation's compute islands guide
eliebakouch · x · 2026-11-03
Core Automation published a long-form piece by Rohan Anil arguing frontier learning algorithms should be co-developed with the world's scattered compute. Key result: they trained a sparse mixture-of-experts model across two clusters of different sizes and GPU types, nearly halfway around the world, over a 25 Gbps bidirectional link — and revisited butterfly mixing, replacing all-reduce with pairwise exchanges between eight learners, matching standard training. The piece argues training compute rarely arrives as one contiguous block; the real bottleneck is power, grid equipment and land: Nadella admits chips sitting unplugged, gas turbines booked through 2028, transformers taking 3+ years, and Goldman Sachs expecting only 50–60% of scheduled US data center capacity to come online on time.
Related event: Ex-Google trainer shows MoE training across 25Gbps transcontinental link(3 posts)→
More from Infra
- Workers IO launches to automate verification, simulating millions of fault-injected production scenarios for AI-written code — ycombinator · 2026-11-03
- Noah Goodman teases "ridiculously fast" weight sync release from his team — gharik · 2026-11-03
- Qualcomm's first High Bandwidth Compute silicon validated on hardware, targeting the memory wall for AI250 — clattner_llvm · 2026-11-03
- DeepSeek-V4.1-Flash 3-bit Quant Hits 93% Accuracy at ~100 tok/s on 256GB M5 Mac Studio — rez0__ · 2026-11-03
- Sampler Pause Stays Under a Second Even Across Distant Datacenters — gharik · 2026-11-03
- Google's Suncatcher satellite runs Gemma 270mi n space—power limits mean only one TPU lit at a time — demian_ai · 2026-11-03