Recurrent Looped Transformer paper: decoder recurs across every token for unbounded latent depth
NandoDF · x · 2026-09-13
A new paper proposes Recurrent Looped Transformer (RLT), an architecture where the decoder runs recurrently across every prompt and response token, carrying its final hidden state and a sliding-window attention cache forward, while a causal encoder builds reusable global KV memory.
Key points:
- Unbounded latent depth: sequences grow computation paths without adding physical layers — a fixed number of blocks per token.
- Three principles: unbounded temporal depth for latent reasoning, model–hardware co-design (separating parallel encoder work from recurrent decoder work, enabling batching, memory reuse, checkpointed training), and model–RL algorithm co-design.
- RL consistency: pretraining, SFT, rollout sampling and current-policy replay share the same state transition, and exact replay reconstructs states under current parameters instead of reusing stale rollout states.
The author positions RLT as a concrete basis for hardware-aware recurrent execution and reliable RL scaling.
More from Infra
- Hugging Bay indexes 149k public open-source AI models with licenses and SHA-256 hashes — johnseach · 2026-09-13
- Southwest Airlines rolls out Starlink Wi-Fi, targeting 300+ aircraft by end of 2026 — XFreeze · 2026-09-13
- Compute is the bottleneck: OpenAI spent millions to crack a Millennium Prize problem — haider1 · 2026-09-13
- $39 Open-Source JetKVM Mini Launches: The Cheapest Way to Give AI Agents Below-OS Machine Control — cheerfulboy · 2026-09-13
- Local LLM model picker: how to choose between Llama, Mistral, Qwen and DeepSeek — anant94 · 2026-09-13
- Nvidia dismisses 'circular financing' concerns, claims every $1 invested returns $100 — mgh2 · 2026-09-13