Looped Transformer's Recurrent Depth: More Reasoning Without Extra Params or Longer CoT
gordic_aleksa · x · 2026-09-17
Aleksa Gordić argues the looped transformer — rumored to be used by frontier labs — deserves more attention as a third scaling dimension: recurrent depth.
Core idea
- Improves reasoning without increasing parameter count or verbalized chain-of-thought length.
- Sharp contrast with MoE: MoE keeps FLOPs constant and adds parameter capacity (knowledge); looping keeps params constant and adds total FLOPs (computation as reasoning).
Architecture
- Computation becomes P → R×r → C: a prelude, a recurrent core repeated r times, and a coda. Crucially, the encoding e is re-injected at every recurrence; hidden state starts from Gaussian noise.
- Training samples r from a heavy-tailed log-normal-Poisson distribution with truncated BPTT, so the model learns across randomized recurrent depths.
Inference scaling
- On ARC-C, performance saturates around 8–12 iterations 0-shot, 20 at 1-shot, and 32 with 25–50 few-shot examples — harder, context-heavy problems benefit from more latent compute. He likens it to learned gradient descent in latent space at inference.
Systems notes
- Sync one r per microbatch across workers to avoid idle GPUs.
- Natural speculative decoding: low r drafts, high r verifies, with draft states reusable (s0→s3 from r=4 continues into r=12).
- Aggressive zero-shot KV cache sharing across recurrence depths since all iterations share weights.
- Easy tokens' latents converge faster, enabling per-token early exit.
- Lower memory in training and inference.
Related event: Looped Transformers Gain Attention as a Way to Scale Test-Time Compute(2 posts)→
More from Research
- DeepMind paper: Veo 3 shows emergent zero-shot reasoning, video models may become vision foundation models — RexDouglass · 2026-09-17
- GoBench: LLMs hit 2500 Elo on 9x9 Go vs KataGo's 4400, r=0.83 with ARC-AGI 2 — Roland31415 · 2026-09-17
- A JSON classifier got 12M views: Niels Rogge visualizes how Jev skips autoregressive generation — rickasaurus · 2026-09-17
- Stanford 2026 AI Index: only half of schools have AI policies, 6% of teachers call them clear — StanfordHAI · 2026-09-17
- AI Researchers Once Put 50% Odds on AI Solving a Millennium Prize Problem by 2054 — Distinct-Question-16 · 2026-09-17
- Scaling Trust Arena Shares Update, Teams with Andon Labs, Publishes Draft Spec — sebkrier · 2026-09-17