Google DeepMind Introduces Recirculation to Improve Transformer State Maintenance
burkov · x · 2026-08-25
Transformers process prompts in parallel, which is efficient but hinders the maintenance of an evolving internal state, often causing correct interpretations formed in deeper layers to be unavailable for timely subsequent processing.
A paper from Google DeepMind and UT Austin proposes a small architectural change called recirculation: at each input step, a portion of the internal representation from a deeper layer is mixed back into a shallower layer in the next step. This creates a recurrent information path without modifying the pretrained model's weights.
Experiments on Gemma models show substantial improvements in language prediction, with an adaptive version reducing perplexity. This suggests that simple architectural tweaks can mitigate Transformer's inherent limitations in state management.
More from Research
- Models lack independent research capabilities; RSI predictions seem overly optimistic — BlancheMinerva · 2026-08-25
- GLiNER 2.5 Launches with Architecture Upgrade for Long-Context Extraction — huggingface · 2026-08-25
- Analysis confirms stealth/ox-alpha is a Z.ai GLM model — PawelHuryn · 2026-08-25
- ProteinDPO aligns protein models for stability, published in Nature Methods — BrianHie · 2026-08-25
- Thinking Machines proposes a safe path for open-weight model releases — luke_drago_ · 2026-08-25
- Headlong experiments with persistent agency via exponential backoff — lateinteraction · 2026-08-25