AI2 releases MoE training tech report: rebuilt Olmo-core scales to 1.2T params on 512 B300 GPUs
AkariAsai · x · 2026-11-18
AI2 researchers (Tianhua Tao, Akari Asai, et al.) published an arXiv tech report, Supercharging Olmo-core for Efficient and Scalable MoE Training.
- Scale: a rebuilt Olmo-core stack supporting MoE models from 12.9B up to 1.2T parameters, training on up to 512 B300 GPUs.
- Problem: MoE adds parameters without raising per-token compute, but training cost also depends on bytes moved, memory held, and kernels launched — costs that track total parameter size. Early baselines saw throughput drop as stored expert parameters grew.
- Key changes: replaced FSDP with a DDP-based architecture; added Expert Parallelism, Pipeline Parallelism, a distributed optimizer, and activation recomputation; proposed NVSHMEM-based rowwise EP that writes tokens directly into expert buffers to skip dispatch/combine permutation overhead; fully synchronization-free EP via GPU-resident routing metadata and device-scheduled grouped GEMMs; tight MXFP8 integration; topology-agnostic checkpoints that allow changing parallel layout between runs.
An interactive web version and a 4-minute video are also available.
More from Infra
- 曝 Nvidia 2022 年牵线阿里与芯片商 Alice Huang,绕道美国出口管制 — teortaxesTex · 2026-11-18
- Microsoft launches Azure AI Gateway Labs: 27 hands-on labs for governing models and agents — adnan_hashmi · 2026-11-18
- vLLM ships Day-0 support for Tencent Hy4 Flash with 8x H200 deployment recipe — vllm_project · 2026-11-18
- SpaceX CFO says power is the real compute bottleneck, eyes top AI infra spot — XFreeze · 2026-11-18
- SGLang hits v1.0 as NVIDIA, AMD, Intel, Google and Meta form its consortium — nikola_mr64990 · 2026-11-18
- Nvidia's CFO walks back defense of 'circular financing' investments — rohanpaul_ai · 2026-11-18