Meta & MIT paper: looping transformer layers beats adding depth at fixed size
rohanpaul_ai · x · 2026-11-02
A new Meta/MIT paper, "How to Loop Your Transformer," systematically studies looping — re-running the same layers multiple times to add depth without extra parameters. Trained from scratch at up to 1B params on 155B tokens (up to 1.07×10²¹ FLOPs, 32 loops), it finds looping the full stack gives the best quality at a fixed parameter budget, while looping a middle block pays off best when training compute is tight. Gains from more loops eventually saturate or reverse; small per-loop low-rank adapters (only +1.1–4.3% params vs +51.4% for doubling depth) help more as loops increase. The paper also fixes Parallel Loop Transformers so loop stages can run concurrently during decoding, removing the linear decoding slowdown.
Related event: Meta and MIT: Looping Transformer Layers Beats Adding Parameters(2 posts)→
More from Research
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02
- New theory paper proves linear decay is the unique minimax-optimal stepsize schedule for SGD — konstmish · 2026-11-02
- Karpathy's nanochat: the best ChatGPT $100 can buy, near 60k GitHub stars — karpathy · 2026-11-02
- After reading 46 resources on agent evals, one author explains why her 95% score was misleading — blaizedsouza · 2026-11-02
- AI tell-tale word 'delve' declining in latest NeurIPS papers, analysis shows — JungWooHa2 · 2026-11-02