DeepLoop: Depth Scaling for Looped Transformers
Shuzhen Li · hf · 2026-07-17
This work discusses the depth scaling problem in Looped Transformers: when the same set of parameters is revisited multiple times, the scaling of residual branches can no longer depend solely on "nominal layers" but must consider parameter access counts.
The authors formalize this tied-depth effect using a first-order perturbation bound and an access alignment coefficient κR, pointing out:
- When accesses are decorrelated, the conclusion reverts to DeepNorm-style exponents
- Under conservative, strongly aligned conditions, as loop counts increase and physical depth remains fixed, the exponent needs to increase from 1/4 to 1/2
Based on this, the DeepLoop method sets new scaling parameters while maintaining the Post-LN DeepNorm architecture:
- α = (2N)^{1/2}
- β = (8N)^{-1/2}
On looped language models at GPT-2 small and GPT-2 medium scales, DeepLoop is largely neutral without repeated physical blocks; but once recursive depth is enabled, it improves validation loss and downstream accuracy. Conclusion: stable recursive depth requires scaling rules designed for "parameter access counts", not just layer counts.
Related event: DeepLoop Explores Depth Scaling for Looped Transformers(2 posts)→
More from Research
- MissionReport benchmark: only 3 of 540 AI-generated government reports pass all criteria — Exp_Mark · 2026-10-27
- Finch 1: a geospatial foundation model trained on imagery from 50+ sensors — Scobleizer · 2026-10-27
- Skyfall AI's SKY-001 world model turns sketches into CAD builds in 3.9s, vs 199s for GPT-6 Astra — rohanpaul_ai · 2026-10-27
- Physicists Let AI Pick Its Own Problem—It Wrote a Publishable Paper in 2 Hours — burny_tech · 2026-10-27
- Researcher flags unresolved data transparency issues in compositionality evals — najoungkim · 2026-10-27
- Separate Forms from Estimators: Why Steer by Difference of Means When You Can Use SGD? — johnhewtt · 2026-10-27