Looping Depth Reshapes Scaling Exponents: 7.4B Model Matches GPT-3 13B with 20x Less Compute

Andrew Gordon Wilson's team released a paper (arXiv 2609.19107) challenging the assumption that architectural interventions only yield constant-factor gains and that scaling exponents can only be changed via data interventions: applying looping (recursive depth) to a standard Transformer during pretraining, combined with model growth and boundary operators, improves the scaling exponent itself, so compute-efficiency gains compound with scale. The current conclusion: architectural innovation can rewrite scaling laws and is complementary to simply adding data and parameters.

Confirmed

• The paper was released by Andrew Gordon Wilson's team, introduced in multiple posts by the authors and team members (m2, m3, m5, m7).
• Core result: a 7.4B looped model matches GPT-3 13B with roughly 20x compute savings.
• Looping acts as a model growth mechanism via recursive deepening; per-scale gain multipliers of about 1.06–1.16 compound with scale (relayed by m1).
• Weight sharing is not compute-optimal on fresh data (paper conclusion relayed by m1).
• Songlin Yang (m6) added commentary: in data-limited, multi-epoch training, loop reuse has a regularization effect; validation loss decreases monotonically with more loops at fixed model size; the optimal number of loops grows with compute budget; compute efficiency improves about 2.2x in that setting.
• The authors note this route is complementary to the conventional data-and-parameter scaling path.

Why it matters

• If scaling exponents can be changed architecturally, the marginal-return structure of compute changes: larger scale saves more compute rather than a one-off constant speedup.
• This offers a new path for efficient pretraining under compute constraints and reopens the debate on whether architectural innovation can rewrite scaling laws.

2026-09-17 ~ 2026-09-19 · 7 related posts

• New Paper: Looping With Model Growth Bends Scaling Laws, Compute Gains Compound — akbirthko · 2026-09-17
• [source] Looping Rewrite Scaling Exponents: 7.4B Model Matches GPT-3 13B With ~20x Less Compute — andrewgwils · 2026-09-17
• New paper: tweaking recursive depth improves pre-training scaling exponents at scale — bo_wangbo · 2026-09-17
• [source] Looping acts as a regularizer, delivering 2.2x compute efficiency over standard Transformers — SonglinYang4 · 2026-09-18
• Looped transformers study: 7.4B growth model matches GPT-3 13B with 20x less compute — burny_tech · 2026-09-19

2 near-duplicate retellings: andrewgwils · burny_tech