Meta's Loop Scaling Laws: looped MoE matches 2x larger models on reasoning
yacinelearning · x · 2026-10-28
Meta AI researchers (Yanbei Chen, Anirudh Goyal, et al.) introduce Loop Scaling Laws, the first scaling law that jointly models recurrence and MoE sparsity alongside model size and data.
- Looped transformers add computational depth via repeated passes; MoE adds capacity via sparse routing — prior laws modeled each axis in isolation
- A bounded, sparsity-conditional recurrence mapping predicts looped-model held-out loss more accurately than alternatives and recovers standard dense/MoE scaling laws as special cases
- The fitted laws give a principled way to design looped MoE under compute/memory budgets: sparsity yields 3x active-parameter efficiency, recurrence 2x total-parameter efficiency on reasoning
- At trillion-token scale, a 1.3B looped MoE matches a 2x larger (2.9B) non-looped MoE at matched training compute, and enables test-time scaling through recurrence
The paper notes looping hits diminishing returns when overused, but sparsity delays the plateau.
More from Models
- Coding agent's compaction task fails with "model is at capacity" error — justalexoki · 2026-10-28
- Nathan Lambert: open models' demise exaggerated, but AA index flattered them — natolambert · 2026-10-28
- TinyLeague drops Artificial Analysis scores, adds KV cache efficiency ratings for local models — nunodonato · 2026-10-28
- Can DeepSeek v4.1 actually run at 1-3 bpw? Reddit locals debate ultra-low quant usability — Plastic-Stress-6468 · 2026-10-28
- Developer pauses his project until Fable 5.5 ships — facontidavide · 2026-10-28
- Hidden model Obsidian 3 tops FrontierCode 2.0 at 46.8%, suspected to be unreleased Fable 5.5 — kimmonismus · 2026-10-28