DeepLoop: Depth Scaling for Looped Transformers
Shuzhen Li · hf · 2026-07-17
This work discusses the depth scaling problem in Looped Transformers: when the same set of parameters is revisited multiple times, the scaling of residual branches can no longer depend solely on "nominal layers" but must consider parameter access counts.
The authors formalize this tied-depth effect using a first-order perturbation bound and an access alignment coefficient κR, pointing out:
- When accesses are decorrelated, the conclusion reverts to DeepNorm-style exponents
- Under conservative, strongly aligned conditions, as loop counts increase and physical depth remains fixed, the exponent needs to increase from 1/4 to 1/2
Based on this, the DeepLoop method sets new scaling parameters while maintaining the Post-LN DeepNorm architecture:
- α = (2N)^{1/2}
- β = (8N)^{-1/2}
On looped language models at GPT-2 small and GPT-2 medium scales, DeepLoop is largely neutral without repeated physical blocks; but once recursive depth is enabled, it improves validation loss and downstream accuracy. Conclusion: stable recursive depth requires scaling rules designed for "parameter access counts", not just layer counts.
Related event: DeepLoop Explores Depth Scaling for Looped Transformers(2 posts)→
More from Research
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11
- Navier-Stokes, Riemann, P vs NP: what this week's math buzzwords mean for you — koltregaskes · 2026-09-11
- Fruit fly brain as an LLM: connectome-driven language model demo goes live — ngxson · 2026-09-11
- Harry Collins: LLMs can't do frontier science because they can't invent new language — whoamisri · 2026-09-11
- The Waymo effect: how AI is quietly making research less collaborative — JohnHammersley · 2026-09-11
- Causal-only attention for non-generative tasks is wasteful, argues HF engineer — antoine_chaffin · 2026-09-11