Rohan Anil's CoreAuto trains MoE across two global clusters over a 25 Gbps link
_arohan_ · x · 2026-11-03
Rohan Anil, whose training algorithms power frontier models, published a CoreAuto write-up on training across "compute islands." His team trained a sparse MoE model across two clusters of different sizes and GPU types, nearly half a world apart, over a bidirectional 25 Gbps link — and matched standard training using butterfly mixing with pairwise exchanges between eight learners instead of all-reduce.
Key arguments:
- Compute rarely arrives as one contiguous block; hundreds-to-thousands-GPU neocloud clusters add up to substantial capacity that learning algorithms should co-develop against.
- The bottleneck is power, grid equipment and land, not chips: Nadella admits chips sitting in inventory that can't be plugged in; gas turbines are booked through 2028, transformers take 3+ years, and Goldman Sachs expects only 50–60% of scheduled US data center capacity online on time (historically 72%).
- A position piece and call to action for truly distributed (eventually in-space) training.
More from Infra
- Mistral Large 4 tech report: 1T-param MoE, 606GB quantized, runs on four GB300 GPUs — _AndrewZhao · 2026-11-03
- Qualcomm hits a significant milestone on its path into the data center — karlfreund · 2026-11-03
- Anthropic's Q3 growth slows to 47% QoQ from 144%, against $895.4B compute commitments — petrusenko_max · 2026-11-03
- vLLM adds Rubin dual-die localization: pairing MoE weight shards with local compute on both dies — anbarth · 2026-11-03
- NVIDIA reportedly producing ~1,000 GB300 racks per week; 1.2M GPUs bound for one campus — acmoytoy · 2026-11-03
- "The VC welfare program is over": Redditor defends OpenAI usage cuts with compute math — R3K4CE · 2026-11-03