How to Loop Your Transformer: Meta/MIT arXiv paper on loop scaling laws
rohanpaul_ai · x · 2026-11-02
Full abstract of Meta FAIR/MIT paper arXiv:2610.40325: looping reuses weights to deepen transformers without extra parameters, and its value depends on whether model size, training compute, or inference efficiency dominates. Experiments (up to 1B params, 155B tokens, 1.07×10²¹ FLOPs, 32 loops) show looping the full stack wins at fixed parameter budget, looping a middle block wins under tight training FLOPs, and gains saturate or reverse beyond some loop count. Small per-loop low-rank adapters improve quality increasingly with more loops. The paper also repairs Parallel Loop Transformers — preserving and separately normalizing the current-token embedding and previous-token hidden state restores strong performance — enabling concurrent loop execution during decoding.
Related event: Meta and MIT: Looping Transformer Layers Beats Adding Parameters(2 posts)→
More from Research
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02
- New theory paper proves linear decay is the unique minimax-optimal stepsize schedule for SGD — konstmish · 2026-11-02
- Karpathy's nanochat: the best ChatGPT $100 can buy, near 60k GitHub stars — karpathy · 2026-11-02
- After reading 46 resources on agent evals, one author explains why her 95% score was misleading — blaizedsouza · 2026-11-02
- Meta & MIT paper: looping transformer layers beats adding depth at fixed size — rohanpaul_ai · 2026-11-02