Looped LM scaling paper: running a recurrent block twice ≈ 1.38× effective parameters
scaling01 · x · 2026-09-02
Highlights the paper "How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models", which studies scaling laws for looped (recurrent depth) language models.
Key implications from the author:
- A related recurrent depth paper found running the same recurrent block twice gives a 1.38× effective-parameter multiplier
- With a Recirculation-style scheme, pre-training compute and decode latency stay unchanged, but inference FLOPs and prefill latency rise
- Back-of-envelope: a 10T-parameter recurrent depth model could perform like a 13.8T model; 7.25T ≈ 10T effective
More from Models
- Models can detect read-only probes with just last-two-layer SFT — Sauers_ · 2026-10-18
- Paying Google AI subscriber blocked: custom-domain Workspace accounts can't access AI Studio — ChrisRackauckas · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18
- Users press OpenAI on long-delayed Adult Mode a year after it was promised — ___Patrice___ · 2026-10-18
- Polymarket opens betting market on who releases the next Gemini Argon model — Polymarket · 2026-10-18
- Grok matter-of-factly lists which groups user was racist against, exempting Africans and Latin Americans — teortaxesTex · 2026-10-18