Frontier training without megacampuses: MoE model trained across <20MW clusters over 25Gbps
_arohan_ · x · 2026-11-05
Arohan endorses Core Automation's piece on training across "compute islands," arguing there is no moat in training — sub-20MW data centers cobbled together can reach frontier today. Key facts: the real bottleneck is power/grid/land, not chips (Nadella: "a bunch of chips sitting in inventory that I can't plug in"); gas turbines booked through 2028, transformers take 3+ years, Goldman expects only 50-60% of planned US DC capacity online on time. The team trained a sparse MoE across two differently-sized clusters with different GPU types, nearly half a world apart, over a 25Gbps link, and matched standard training using pairwise butterfly mixing instead of all-reduce.
More from Infra
- US AI scale faces two physical gates: delivered power and manufacturing — ingliguori · 2026-11-05
- Liquid AI ships LFM 3 models (1.7B/2B) running locally on iPhone Duo via MLX — helloiamleonie · 2026-11-05
- Yuma backs Bittensor subnet Fugal claiming GPT-5.5-level results at 1/6 the cost — markjeffrey · 2026-11-05
- Speculative decoding makes Qwen3.6/3.8 much faster, latest draft model adds ~10% — teortaxesTex · 2026-11-05
- webAI open-sources 8.8B TwIL-8B hitting 93% of frontier coding performance — dr_cintas · 2026-11-05
- Together Compiler optimizes new open models for production in hours, not weeks — togethercompute · 2026-11-05