Ai2 Open-Sources Olmo-core 3, a Trillion-Parameter MoE Training Stack
On October 1, Allen Institute for AI (Ai2) released and open-sourced Olmo-core 3, the core training system for the next-generation OLMo, designed to scale MoE (mixture-of-experts) training to trillion-parameter scale while keeping compute efficient, continuing Ai2's commitment to open-sourcing all training tools and infrastructure.
Confirmed
- Olmo-core 3 is a PyTorch training stack for trillion-parameter MoE models, Apache-2.0 licensed, open-sourced on GitHub (allenai/OLMo-core) and published on PyPI (ai2-olmo-core)
- In experiments, scaling the expert pool from 8 to 128 while keeping 3.2B active parameters reduced training throughput by under 5%
- Overall training throughput improved 2.7x over the previous version
- Ai2 also shipped a technical report and an interactive explainer, "How do 512 GPUs train one AI model?", breaking down single-GPU compute/memory, data parallelism, memory sharding and expert parallelism, and how the three parallelism strategies combine to support large MoE training
- Ai2 noted in the explainer that load-balancing scores can be "deceiving", not truly reflecting actual load distribution during scaling
Why it matters
- Olmo-core 3 provides a fully open, reproducible and auditable foundation for trillion-parameter MoE training, contrasting with closed-source training stacks
- The measured figures—under 5% throughput loss at 128 experts and 2.7x overall throughput gain—offer concrete reference points for MoE efficiency at extreme scale
- The accompanying 512-GPU interactive explainer opens up usually-hidden large-scale parallelism engineering details in tutorial form, valuable to researchers and engineering teams alike
2026-10-01 ~ 2026-10-03 · 5 related posts
Primary sources
- Ai2 open-sources Olmo-core 3, training infrastructure scaling MoE models toward trillion parameters — allen_ai ·
- Ai2's tech report shows how 512 GPUs train one model—and flags 'token gerrymandering' — allen_ai ·
- Ai2 Releases Olmo-core 3: Open MoE Training Framework Scales to 128 Experts with <5% Throughput Loss — allen_ai ·
- [source] Ai2 open-sources Olmo-core 3, training infrastructure scaling MoE models toward trillion parameters — allen_ai · 2026-10-01
- [source] Ai2's tech report shows how 512 GPUs train one model—and flags 'token gerrymandering' — allen_ai · 2026-10-01
- [source] Ai2 Releases Olmo-core 3: Open MoE Training Framework Scales to 128 Experts with <5% Throughput Loss — allen_ai · 2026-10-01
- AI2 open-sources Olmo-core 3, a PyTorch MoE training stack with 2.7x throughput gains — lmoroney · 2026-10-03
1 near-duplicate retellings: lmoroney