Recurrent Looped Transformer extends looped architecture across tokens, adding depth per token
omarsar0 · x · 2026-09-14
A new technical report extends the popular looped transformer idea across tokens. Recurrent Looped Transformer (RLT) makes the decoder recurrent over every token, prompt and response included: a causal encoder builds global KV memory, and for each new token the decoder combines the encoder representation, its own previous final hidden state, and a sliding-window cache of recent activations. With a 48-layer decoder, the compute path after t tokens runs through 48t blocks while per-token cost stays fixed. The same state transition serves pretraining, SFT, sampling and RL replay, with no reset at the prompt-response boundary; RL replay rebuilds states under current weights instead of reusing stale rollouts.
More from Models
- Fable 5.5 Max stuns with gorgeous SVG of a pelican riding a bicycle, headed to Claude Code — ChrisGPT · 2026-10-31
- OpenAI reportedly to train on users' Daybreak sessions with no opt-out — dyn___ · 2026-10-31
- NVIDIA Nemotron 3.5 scales LLM self-verification, +9-10 points on agentic benchmarks — sarahcat21 · 2026-10-31
- Mistral Large 4 scores 94% on 1,273 USMLE-style questions but flips 44 at temp 0 — MaziyarPanahi · 2026-10-31
- Researchers find 5 personality structures in LLM weight space, Big Five-like — luke_drago_ · 2026-10-31
- OpenAI cuts individual API access to Daybreak Blue starting November 20 — mkosmo · 2026-10-31