Loop Scaling Laws: First Scaling Law Jointly Modeling Recurrence and MoE Sparsity, Promising ~2x Parameter Savings
anirudhg9119 · x · 2026-10-02
An arXiv paper introduces Loop Scaling Laws, the first scaling law to jointly model recurrence, MoE sparsity, model size, and data. A bounded, sparsity-conditional recurrence mapping characterizes the effective-parameter gain from looping and how sparsity raises that ceiling; the fitted laws predict held-out loss of looped models more accurately than prior alternatives and recover standard dense and MoE scaling laws as special cases.
Key findings:
- Sparsity delivers 3x active-parameter efficiency; recurrence yields 2x total-parameter efficiency on reasoning; the two axes are complementary.
- The laws give a principled basis for designing looped MoE models under compute and memory constraints, enabling test-time scaling via recurrence.
- Gains hold at trillion-token scale: at matched compute, a looped MoE with law-derived recurrence matches a 2x larger non-looped MoE on reasoning benchmarks.
Authors: Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi (19 pages).
Related event: Meta Unveils First Scaling Laws for Looped Mixture of Experts(3 posts)→
More from Models
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- Stealth Anthropic model spotted in Devin build as METR chart shows frontier pace up 1.4x — daniel_mac8 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- Unsloth Q8 crushes default Q3 in a self-hosted test over 331 financial PDFs — No_Cartographer3953 · 2026-11-16
- METR time-horizon chart updated: doubling every 1.7 months since Opus 4.5, 'Model 4' ~38h — ben_j_todd · 2026-11-16