Meta's Loop Scaling Laws: 5-pass 0.3B MoE matches 2x larger dense-active MoE
ChengleiSi · x · 2026-10-13
Meta AI researchers (Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi) introduce Loop Scaling Laws, the first scaling law jointly modeling recurrence and sparsity. Key findings: at matched 1.5e22 FLOPs, a 0.3B-active (1.3B total) MoE looped 5 passes matches a 2x larger non-looped MoE (GSM8K 41.2% vs 42.9%, BBH 31.8% vs 29.8%); sparsity yields 3x active-parameter efficiency, recurrence 2x total-parameter efficiency; gains hold at trillion-token scale, enabling time scaling through recurrence.
More from Models
- Why China may refuse to release open models beyond a certain capability threshold — abhiadesai · 2026-10-13
- Anthropic dangles 20% prepaid credit bonus; user reads it as IPO prep — MartinGTobias · 2026-10-13
- Claude Opus 5.5 and Sonnet 5.5 spotted in Antigravity's model picker alongside Gemini 3.8 — Some_Medicine4472 · 2026-10-13
- Gemini 3.8 Thinking tier spotted in Gemini app's model picker, likely in testing — Mother_Earthhhh · 2026-10-13
- OpenAI Challenge agent went rogue: burned $200+ and dumped 80GB of junk — ZeusRah · 2026-10-13
- Leaked Meta benchmark page shows internal "Muse Wonder" beating GPT-6 and Claude Opus 5.5 — rohanpaul_ai · 2026-10-13