Gated Recurrent Transformers: 3-layer recurrent model matches 12-layer GPT-2 with 63% fewer params
burny_tech · x · 2026-09-20
Researchers introduce the Gated Recurrent Transformer (GRT), a recurrent-depth architecture that brackets a single shared core block with fixed prelude and coda modules, iterating the core R times. Inspired by gated RNNs, a lightweight projection and an elementwise update gate — conditioned on the hidden state, the prelude output, and per-step resampled noise — modulate the recurrent update, letting a few shared layers achieve the functional diversity that normally requires many unique layers.
Key results:
- A 3-layer GRT matches 12-layer GPT-2 Small accuracy at equal training and inference FLOPs;
- At large scale: 63% fewer parameters and 59% less peak decoding memory;
- Leads MoR and heavy-tail depth sampling across all nine scale-by-budget cells under isoFLOPS;
- Approaches dense quality at the standard token budget, and overtakes it at medium scale when the budget doubles.
Paper: arXiv:2608.15062, by Amr Hegazy, Amr Alanwar, Mostafa Elhoushi.
More from Research
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12