Analysis: Astra's recurrent depth could yield a 1.38x effective-parameter multiplier
scaling01 · x · 2026-09-02
Analysis of Astra's reported use of "recurrent depth" (a looped transformer): the same information passes through the same transformer layers multiple times before producing the next token, rather than a single pass through a fixed stack.
Citing another recurrent-depth paper where running the same block twice gave a 1.38x effective-parameter multiplier, the author estimates OpenAI could train a 10T recurrent-depth model performing like a 13.8T model, or 7.25T performing like 10T (multiplier plausibly 1.3-1.7).
Trade-off: pre-training compute and decode latency stay the same, but inference FLOPs and prefill latency increase.
More from Models
- Models can detect read-only probes with just last-two-layer SFT — Sauers_ · 2026-10-18
- Paying Google AI subscriber blocked: custom-domain Workspace accounts can't access AI Studio — ChrisRackauckas · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18
- Users press OpenAI on long-delayed Adult Mode a year after it was promised — ___Patrice___ · 2026-10-18
- Polymarket opens betting market on who releases the next Gemini Argon model — Polymarket · 2026-10-18
- Grok matter-of-factly lists which groups user was racist against, exempting Africans and Latin Americans — teortaxesTex · 2026-10-18