1T-Parameter MoE trained on EU public supercomputer with PyTorch, per-GPU throughput up 14x
PyTorch · x · 2026-11-12
The EUROPA consortium (Domyn + Fraunhofer IAIS/IIS) scaled pretraining of a 1T-parameter MoE on JUPITER, EuroHPC's shared exascale supercomputer at Forschungszentrum Jülich, using PyTorch and TorchTitan — boosting per-GPU throughput from 90 to 1,300 tokens/s across 384 GH200 GPUs per replica (96 nodes).
The model has 951.9B total parameters (32.8B active per token), 74 layers, 384 routed experts with top-8, a 256K vocabulary, 65,536-token sequences, and uses Muon + AdamW. Part of the EuroHPC Frontier AI Grand Challenge, the project aims to train an open European MoE of at least 400B parameters covering all 24 official EU languages in compliance with the EU AI Act — proving shared public infrastructure can sustain frontier-scale training.
More from Infra
- SpaceX's Southaven data centers net local schools and county $63M a year — 0xAllen_ · 2026-11-12
- Quantum infrastructure firm Q-CTRL announces Series C led by Salesforce Ventures — MJBiercuk · 2026-11-12
- Voxel pagoda generated fully local on a GPU-less 2020 ThinkPad with a 35B MoE model — dai_app · 2026-11-12
- Nebius Q3: $12B in customer commitments, revenue up 68% to $981M as GPU prices climb 22% — demian_ai · 2026-11-12
- llama.cpp crosses 134k GitHub stars as the default local LLM inference stack — ggml-org · 2026-11-12
- TensorFold: a Zig LLM inference engine spanning Metal, CUDA and Vulkan — ashhart · 2026-11-12