Looped Transformer's Recurrent Depth: More Reasoning Without Extra Params or Longer CoT
gordic_aleksa · x · 2026-09-17
Aleksa Gordić argues the looped transformer — rumored to be used by frontier labs — deserves more attention as a third scaling dimension: recurrent depth.
Core idea
- Improves reasoning without increasing parameter count or verbalized chain-of-thought length.
- Sharp contrast with MoE: MoE keeps FLOPs constant and adds parameter capacity (knowledge); looping keeps params constant and adds total FLOPs (computation as reasoning).
Architecture
- Computation becomes P → R×r → C: a prelude, a recurrent core repeated r times, and a coda. Crucially, the encoding e is re-injected at every recurrence; hidden state starts from Gaussian noise.
- Training samples r from a heavy-tailed log-normal-Poisson distribution with truncated BPTT, so the model learns across randomized recurrent depths.
Inference scaling
- On ARC-C, performance saturates around 8–12 iterations 0-shot, 20 at 1-shot, and 32 with 25–50 few-shot examples — harder, context-heavy problems benefit from more latent compute. He likens it to learned gradient descent in latent space at inference.
Systems notes
- Sync one r per microbatch across workers to avoid idle GPUs.
- Natural speculative decoding: low r drafts, high r verifies, with draft states reusable (s0→s3 from r=4 continues into r=12).
- Aggressive zero-shot KV cache sharing across recurrence depths since all iterations share weights.
- Easy tokens' latents converge faster, enabling per-token early exit.
- Lower memory in training and inference.
Related event: Looped Transformers Gain Attention as a Way to Scale Test-Time Compute(2 posts)→
More from Research
- Radial gate objective cuts diffusion transformer FID from 16.24 to 11.83 — Junha Park · 2026-11-02
- REA: async brain-cerebellum runtime for robots where policies never wait on the LLM — Pengfei Zhou · 2026-11-02
- Microsoft's MemSteer turns unlabeled user histories into agent memory supervision — MicrosoftResearch · 2026-11-02
- SciGame benchmark uses counterfactual worlds to expose LLM agents' priors vs evidence — Yusheng Zhao · 2026-11-02
- On Terence Tao's blog, Frank Calegari calls on mathematicians to adapt after OpenAI's 700-paper release — Hesamation · 2026-11-02
- Tsinghua & Microsoft's HarnessMM cuts agent token use by up to 87.93% while raising success rate — rohanpaul_ai · 2026-11-02