Recirculation paper: leaking late-layer activations boosts frozen Gemma3's GSM8k accuracy by 21%
s_scardapane · x · 2026-09-10
A new arXiv paper by Michael Mozer et al. introduces Recirculation, an inference-time architectural enhancement that "leaks" late-layer activations back into early layers, giving feedforward transformers a form of recurrence for tracking belief states.
Key points:
- Near-zero extra latency at generation; only prefill requires serial processing
- Complements chain-of-thought: CoT for complex inference, recirculation for basic state tracking; distinct from depth-looping and costly recurrent-transformer training
- An adaptive variant requires no weight training, just light hyperparameter tuning with the base model frozen
- On the Gemma3 family it systematically reduces perplexity across datasets and improves GSM8k accuracy by 21%, with reliable gains on other tasks
The authors frame this as a route to architectural evolution guided by a trained network's own properties.
More from Research
- PKU paper: rewriting influential training samples shifts model behavior more than reweighting — PKU · 2026-09-10
- StochBench: 450 graduate-level stochastic process problems in Lean 4 for theorem proving — Idan Davidovich · 2026-09-10
- KaRMA: first metric to quantify robot hand dexterity from kinematics alone, at IROS 2026 — pulkitology · 2026-09-10
- Ego-Exo 4D dataset highlighted as key to closing the 4D understanding gap — ducha_aiki · 2026-09-10
- Lean gets con-leche: an external checker proven consistent, never accepts a proof of False — AlexKontorovich · 2026-09-10
- Kristen Grauman talks machine perception to human expertise at ECCV 2026 — ducha_aiki · 2026-09-10