GRT Architecture: Halves Params to Match GPT-2 Performance
burny_tech · x · 2026-08-30
The paper proposes 'Gated Recurrent Transformers (GRT)', decoupling depth from parameter count by reusing a small shared Transformer core. It uses a gate conditioned on the hidden state to modulate updates, allowing the same weights to exhibit different behaviors at different depths. Experiments show a 3-layer GRT matches a 12-layer GPT-2 Small under isoFLOPs. Large models reduce parameters by 62% and peak decoding memory by 59%. Recurrence depth also serves as an inference-time compute hyperparameter.
Related event: Gated Recurrent Transformer Matches GPT-2 with Far Fewer Parameters(2 posts)→
More from Research
- A beginner-friendly introduction to causal inference kicks off, Part 1 — Nasereliver · 2026-10-16
- FBI method trains for Pass@K with one sample per update at 12.3% of the cost — EmtiyazKhan · 2026-10-16
- Berkeley researchers sample humanoid robot policies directly from weight space with diffusion — berkeley_ai · 2026-10-16
- OneWM: unified causal training shows world and action modeling improve each other — IAAI233 · 2026-10-16
- Baidu's Virtual Head Attention tames the KV cache, FLOPs, quality triangle in long-context LLMs — baidu · 2026-10-16
- 5.8M-param Speech Event Detector adds laughs, breaths and背景 sounds to Whisper transcripts — Ayush Kumar Bar · 2026-10-16