AI2 open-sources Olmo-core 3, a PyTorch MoE training stack with 2.7x throughput gains

lmoroney · x · 2026-10-03

Allen Institute for AI released Olmo-core 3 (Apache-2.0), a PyTorch training stack for MoE models up to trillion-parameter scale. Growing the expert pool from 8 to 128 at 3.2B active params cost under 5% throughput; on eight B300 GPUs it delivers 2.7x tokens/sec vs their earlier FSDP-based MoE path. A useful open case study in expert parallelism, routing costs, and sparse-model efficiency.

Related event: Ai2 Open-Sources Olmo-core 3, a Trillion-Parameter MoE Training Stack(5 posts)→

Original post →

More from Infra

Infra channel →