Recursive depth looping improves scaling exponents: 7.4B model matches GPT-3 13B with 20x less compute

andrewgwils · x · 2026-09-17

A new paper from Andrew Wilson's group shows that looping (recursive depth growth) during pre-training can improve the scaling exponent itself—not just constants—challenging the assumption that only data interventions move scaling exponents.

Method: grow model depth by increasing loop count during training (tied and untied weights), plus a simple boundary operator that normalizes the residual stream and injects an earlier block.
Results: a 7.4B model-growth architecture matches GPT-3 13B on CORE with roughly 20x less compute.
Rationale: effective depth should be maximized at any compute budget; the "curse of depth" often makes later blocks contribute vanishingly little.
In multi-epoch training, standard looping is compute-optimal with the optimal loop count rising with budget, acting as a useful regularizer; effective depth is key in both single- and multi-epoch regimes.

Original post →

Recursive depth looping improves scaling exponents: 7.4B model matches GPT-3 13B with 20x less compute

More from Models