Gated Recurrent Transformers: 3-layer recurrent model matches 12-layer GPT-2 with 63% fewer params
burny_tech · x · 2026-09-20
Researchers introduce the Gated Recurrent Transformer (GRT), a recurrent-depth architecture that brackets a single shared core block with fixed prelude and coda modules, iterating the core R times. Inspired by gated RNNs, a lightweight projection and an elementwise update gate — conditioned on the hidden state, the prelude output, and per-step resampled noise — modulate the recurrent update, letting a few shared layers achieve the functional diversity that normally requires many unique layers.
Key results:
- A 3-layer GRT matches 12-layer GPT-2 Small accuracy at equal training and inference FLOPs;
- At large scale: 63% fewer parameters and 59% less peak decoding memory;
- Leads MoR and heavy-tail depth sampling across all nine scale-by-budget cells under isoFLOPS;
- Approaches dense quality at the standard token budget, and overtakes it at medium scale when the budget doubles.
Paper: arXiv:2608.15062, by Amr Hegazy, Amr Alanwar, Mostafa Elhoushi.
More from Research
- ianand: Jev isn't a ChatGPT replacement but a new 'decision AI' tool for devs — ianand · 2026-09-20
- ianand: Jev gives the Encoder a ChatGPT-equivalent product surface — ianand · 2026-09-20
- In Jev's parallel universe, AI is faster and reliable but only makes decisions — ianand · 2026-09-20
- ChatGPT, Claude, Gemini all descend from GPT-2's decoder-only line, ianand explains — ianand · 2026-09-20
- The parallel universe with no GPT-2 moment where innovation went to the Encoder — ianand · 2026-09-20
- ianand's thread: Jev, an Encoder-based AI model from a parallel universe — ianand · 2026-09-20