Google DeepMind Introduces Recirculation to Improve Transformer State Maintenance
burkov · x · 2026-08-25
Transformers process prompts in parallel, which is efficient but hinders the maintenance of an evolving internal state, often causing correct interpretations formed in deeper layers to be unavailable for timely subsequent processing.
A paper from Google DeepMind and UT Austin proposes a small architectural change called recirculation: at each input step, a portion of the internal representation from a deeper layer is mixed back into a shallower layer in the next step. This creates a recurrent information path without modifying the pretrained model's weights.
Experiments on Gemma models show substantial improvements in language prediction, with an adaptive version reducing perplexity. This suggests that simple architectural tweaks can mitigate Transformer's inherent limitations in state management.
Related event: DeepMind's Recirculation Evolves Transformers Without Retraining(3 posts)→
More from Research
- Cohere Labs world model course session 4: DreamerV1-V4 architectures in one go — Cohere_Labs · 2026-10-16
- Reasonable post-trains Nemotron 3.5 on 3,000 Verus proofs, lifting pass@1 ~30 points to 73.3% — fhuszar · 2026-10-16
- GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently — nitarshan · 2026-10-16
- One tiny word changes the conversation: how LLMs pick up on 'just' — StellaLisy · 2026-10-16
- SEAR: first large-scale controlled study of self-evolving robot agents, 7 LLMs, 330 tasks — xiye_nlp · 2026-10-16
- After Claude-assisted elliptic curve records fell, EC Rank challenge opens 156 record slots — Jsevillamol · 2026-10-16