Frontier training algo author trains MoE across continents on 25 Gbps
andersonbcdefg · x · 2026-11-03
Rohan Anil, developer of training algorithms behind frontier models, published an 18-minute writeup from CoreAuto: they trained a sparse 3B MoE across two differently-sized clusters with different GPUs nearly halfway around the world over a 25 Gbps bidirectional link. They also replaced all-reduce with pairwise butterfly mixing between eight learners, matching standard training quality. The piece is both a position paper and a call to action for truly distributed learning, eyeing compute islands and in-space training.
Related event: Rohan Anil shows MoE training across continents over 25Gbps links(6 posts)→
More from Infra
- Gary Marcus predicts the data center binge will collapse, with many bankruptcies ahead — GaryMarcus · 2026-11-05
- Merge's eval claims its MCP connectors beat Microsoft's native servers on tool count and reliability — shensi · 2026-11-05
- TorchTPU compat study: 4,821 of 4,917 community models run crash-free on TPUs — jiqizhixin · 2026-11-05
- Foxconn targets 50% of global AI server market, up from ~40% today — Beth_Kindig · 2026-11-05
- antirez: prefetching MoE experts via previous-layer router lets K3.1 stream on a 128GB box — antirez · 2026-11-05
- Decentralized RL Works: Untrusted Miners Lift LiveCodeBench 39.5→58.7 on Bittensor Subnet — bittingthembits · 2026-11-05