LOOM Scales Looped MoE to 9-12 Loops, Cutting 1.7B Model Perplexity From 9.62 to 7.77

Intelligent-Systems · hf · 2026-10-05

A new paper diagnoses why looped mixture-of-experts LLMs stop benefiting beyond two loops and proposes LOOM as a scalable recipe.

Two obstacles: looping amplifies the curse of depth (hidden-state variance grows each iteration, destabilizing recurrence), and expert selection collapse (routers repeatedly pick the same experts, so extra loops add compute without diversity).
LOOM bounds variance by scaling residual updates and re-injecting the input embedding every loop, while per-loop routers and a Looping Residual diversify computation.
Experiments across 100M-1.7B models scale stably to 9-12 loops: under near-iso-FLOP, the 700M model peaks at 5 loops (perplexity 18.36→16.54, zero-shot accuracy 38.84%→39.53%); without FLOP matching, the 1.7B model on 60B tokens peaks at 9 loops (perplexity 9.62→7.77, accuracy 42.4%→47.7%). Code is open-sourced.

More from Research