A $100M compute study says layer-loop beats model-loop for large MoE pretraining
teortaxesTex · x · 2026-07-21
The first author says the team spent hundreds of millions of dollars in compute validating the classical Universal Transformer, or “model-loop,” setup. They found that model-loop does not scale well to billion-parameter MoE models. In later-stage pretraining, their “layer-loop” approach outperformed model-loop, stayed ahead through the end of training, and was friendlier to infrastructure and pipeline parallelism. They also argue that, at equal theoretical FLOPs, the looped approach can beat non-loop baselines, and that model-loop places parameters and gradients farther apart, which may slow training as scale grows.
Related event: Loopie Cyclic Transformer Beats 30B Baseline with Equal Compute(6 posts)→
More from Infra
- AI performance is increasingly limited by materials science, not just compute — nordicinst · 2026-07-21
- A GLM-5.2 inference debate asks how 750B parameters can exceed 1 token per second — francoisfleuret · 2026-07-21
- Why vector databases slow AI agents down after constant writes — PrajwalTomar_ · 2026-07-21
- Larry Fink says China is ahead in the AI energy race, citing 100 GW nuclear buildout — rohanpaul_ai · 2026-07-21
- Local AI may pay back in 6–7 years and cut long-term costs by 30–40% — DavidLinthicum · 2026-07-21
- TSMC reportedly plans up to 10% chipmaking price hikes in 2027 — kimmonismus · 2026-07-21