RLT: Recurrent Looped Transformer Goes Open Source With Cross-Token Recursion
Yifan Zhang released a new architecture, Recurrent Looped Transformer (RLT), with open code and paper. It extends popular looped transformers from looping layers to looping tokens, addressing standard Transformers' weaknesses on long-range computation.
Confirmed
- RLT achieves latent-space reasoning with unbounded depth; network depth grows linearly with sequence length.
- Architecture: a causal encoder builds global KV memory, and a recurrent decoder combines it with sliding-window attention (SWA), recursing over each prompt and response token, with the final hidden state of one token feeding the next.
- @techNmak relayed paper figures: a 48-layer Transformer's recurrent state forms a path of 48,000 blocks after 1,000 tokens.
- @burkov noted the motivation that existing models reset state at the prompt-response boundary, limiting long-range computation.
- @rasbt noted similarity to DeepSeek's encoder-decoder global KV cache plus looped transformer design.
- Code and paper are open-sourced on GitHub.
Why it matters
- Cross-token recurrence removes the fixed-depth limitation, offering a new path for tasks requiring long-range reasoning; echoes of DeepSeek's design suggest growing industry interest in this direction.
2026-09-13 ~ 2026-09-15 · 6 related posts
Primary sources
- Recurrent Looped Transformer extends looped architecture across tokens, adding depth per token — omarsar0 ·
- Recurrent Looped Transformer: new architecture adds unbounded temporal depth for RL-friendly compute — burkov ·
- Recurrent Looped Transformer report: looped decoding with global KV memory — rasbt ·
- Recurrent Looped Transformer (RLT) open-sourced: latent reasoning with infinite depth — inductionheads · 2026-09-13
- Recurrent Looped Transformer paper: decoder recurs across every token for unbounded latent depth — NandoDF · 2026-09-13
- [source] Recurrent Looped Transformer extends looped architecture across tokens, adding depth per token — omarsar0 · 2026-09-14
- [source] Recurrent Looped Transformer report: looped decoding with global KV memory — rasbt · 2026-09-14
- [source] Recurrent Looped Transformer: new architecture adds unbounded temporal depth for RL-friendly compute — burkov · 2026-09-14
- Recurrent Looped Transformer paper: recurrent state hits 48,000-block path after 1,000 tokens — techNmak · 2026-09-14