Recurrent Looped Transformer paper: decoder recurs across every token for unbounded latent depth
NandoDF · x · 2026-09-13
A new paper proposes Recurrent Looped Transformer (RLT), an architecture where the decoder runs recurrently across every prompt and response token, carrying its final hidden state and a sliding-window attention cache forward, while a causal encoder builds reusable global KV memory.
Key points:
- Unbounded latent depth: sequences grow computation paths without adding physical layers — a fixed number of blocks per token.
- Three principles: unbounded temporal depth for latent reasoning, model–hardware co-design (separating parallel encoder work from recurrent decoder work, enabling batching, memory reuse, checkpointed training), and model–RL algorithm co-design.
- RL consistency: pretraining, SFT, rollout sampling and current-policy replay share the same state transition, and exact replay reconstructs states under current parameters instead of reusing stale rollout states.
The author positions RLT as a concrete basis for hardware-aware recurrent execution and reliable RL scaling.
More from Infra
- Clover-35B-A3B Quantized to 29.4 GB Runs at ~42 tok/s on a 64 GB Mac — viveky259 · 2026-10-31
- Liquid Agent brings fully local LFM3 agent to Linux with one-command install — JosephJacks_ · 2026-10-31
- A fully local office AI stack: MiniCPM, Qwen3, LFM2.5 and Whisper — HowDevelop · 2026-10-31
- a16z: GPU rental prices up 106% in 3 months while token prices fall 58% — a16z · 2026-10-31
- vLLM nightly adds PLE disk offload, with a telling [Qwen4Exp] tag on the PR — rmhubbert · 2026-10-31
- Solo AI studio: per-minute GPU rentals rendered a 328-frame Blender shot for ~$1 — r0ck3t23 · 2026-10-31