DeepMind Paper: Leaking Activations Back Fixes Long-Context State Loss
rohanpaul_ai · x · 2026-08-25
A new Google DeepMind paper proposes 'Recirculation' to address state loss in long inputs. Since shallow layers miss context worked out by deep layers, this method mixes a fraction of deep layer activations back into shallow layers during inference without retraining. On Gemma 3, perplexity dropped on 9/10 datasets (up to 35% for the 12B model), though multiple-choice gains were modest.
Related event: DeepMind's Recirculation Evolves Transformers Without Retraining(3 posts)→
More from Research
- Interpretability debate: 'features' as fundamental neural network units may be wrong — MoonL88537 · 2026-10-16
- FOMO-Dex adds force-aware compliance to dexterous teleoperation for contact-rich robot tasks — HaozhiQ · 2026-10-16
- New mech interp hypothesis: neural networks generalize from the very first training step — burny_tech · 2026-10-16
- Princeton launches research initiative on keeping autonomous AI systems aligned with human intent — LakeBrenden · 2026-10-16
- OrganicHAR at UbiComp 2026: Discovering Activities Sensors Can Actually Distinguish — mynkgoel · 2026-10-16
- Dev critiques compiling fast programs into slow neural nets: fun, but not useful — ctjlewis · 2026-10-16