GRT Architecture: Halves Params to Match GPT-2 Performance
burny_tech · x · 2026-08-30
The paper proposes 'Gated Recurrent Transformers (GRT)', decoupling depth from parameter count by reusing a small shared Transformer core. It uses a gate conditioned on the hidden state to modulate updates, allowing the same weights to exhibit different behaviors at different depths. Experiments show a 3-layer GRT matches a 12-layer GPT-2 Small under isoFLOPs. Large models reduce parameters by 62% and peak decoding memory by 59%. Recurrence depth also serves as an inference-time compute hyperparameter.
Related event: Gated Recurrent Transformer Matches GPT-2 with Far Fewer Parameters(2 posts)→
More from Research
- Google Paper: Autonomous AI Research Hallucinates 90% Without Checks — rohanpaul_ai · 2026-09-01
- RLHF impact on tokens: unconscious shifts vs conscious choices — voooooogel · 2026-09-01
- On token layers and consciousness in RLHF — voooooogel · 2026-09-01
- CommerceAgentBench released: Qwen leads open-weight models — Alibaba_Qwen · 2026-09-01
- Discussion on Why Universal Time Series Models Work — Afinetheorem · 2026-09-01
- New paper: a structured ladder for scaling large reasoning models beyond human supervision — Zhiqin Yang · 2026-09-01