Recursive depth looping improves scaling exponents: 7.4B model matches GPT-3 13B with 20x less compute
andrewgwils · x · 2026-09-17
A new paper from Andrew Wilson's group shows that looping (recursive depth growth) during pre-training can improve the scaling exponent itself—not just constants—challenging the assumption that only data interventions move scaling exponents.
- Method: grow model depth by increasing loop count during training (tied and untied weights), plus a simple boundary operator that normalizes the residual stream and injects an earlier block.
- Results: a 7.4B model-growth architecture matches GPT-3 13B on CORE with roughly 20x less compute.
- Rationale: effective depth should be maximized at any compute budget; the "curse of depth" often makes later blocks contribute vanishingly little.
- In multi-epoch training, standard looping is compute-optimal with the optimal loop count rising with budget, acting as a useful regularizer; effective depth is key in both single- and multi-epoch regimes.
Related event: Looped Depth Improves Scaling Exponents, 7.4B Model Matches GPT-3 13B(4 posts)→
More from Models
- Using an LLM as benchmark scorer fails: over-optimistic ratings diverge from human judgment — amplifiedamp · 2026-09-18
- Jev as an LLM judge flops: scores nearly everything positively, disagrees with humans — amplifiedamp · 2026-09-18
- Noam Brown: models may perform their chain of thought; alignment must be solved — infoxiao · 2026-09-18
- Jev fails as an LLM scorer on OntBench: rates almost everything positively, contradicting human and Codex ratings — amplifiedamp · 2026-09-18
- Independent eval puts new model Jev at Terra no-think level, roughly on par with Luna-xhigh — tokenbender · 2026-09-18
- Hugging Face adds zero-shot text classification pipeline to its API, no training needed — joeddav · 2026-09-18