Looped transformers study: 7.4B growth model matches GPT-3 13B with 20x less compute

burny_tech · x · 2026-09-19

arXiv paper 2609.19107 shows architectural interventions can change pre-training scaling exponents:

Weight sharing is not compute-optimal on fresh data, costing a constant 1.06–1.16x compute at every scale; its benefit only appears in data-constrained, multi-epoch training.
The gains of looping come from the norm + input re-injection boundary operator and growing depth mid-training — both improve the scaling exponent without weight sharing.
Model growth yields the biggest exponent shifts: a 7.4B growth architecture matches GPT-3 13B on CORE with roughly 20x less compute, with efficiency gains that increase with scale.
In multi-epoch settings, looping regularizes, and it is compute-optimal to increase loop count with scale.

More from Infra