DeepLoop: Depth Scaling for Looped Transformers
Shuzhen Li · hf · 2026-07-17
This work discusses the depth scaling problem in **Looped Transformers**: when the same set of parameters is revisited multiple times, the scaling of residual branches can no longer depend solely on "nominal layers" but must consider parameter access counts. The authors formalize this tied-depth effect using a first-order perturbation bound and an access alignment coefficient **κ_R**, pointing out: - When accesses are decorrelated, the conclusion reverts to DeepNorm-style exponents - Under conservative, strongly aligned conditions, as loop counts increase and physical depth remains fixed, the exponent needs to increase from **1/4** to **1/2** Based on this, the **DeepLoop** method sets new scaling parameters while maintaining the Post-LN DeepNorm architecture: - **α = (2N)^{1/2}** - **β = (8N)^{-1/2}** On looped language models at GPT-2 small and GPT-2 medium scales, DeepLoop is largely neutral without repeated physical blocks; but once recursive depth is enabled, it improves validation loss and downstream accuracy. Conclusion: stable recursive depth requires scaling rules designed for "parameter access counts", not just layer counts.
Related event: DeepLoop Explores Depth Scaling for Looped Transformers(2 posts)→
More from Research
- LTX 2.3 LoRA demo changes a video’s camera angle — CQDSN · 2026-07-21
- OpenForecaster uses daily news to improve language-model forecasting — Cohere_Labs · 2026-07-21
- SenseTime unveils U1 Pro and open-sources a 50M-sample vision dataset at WAIC 2026 — 机器之心 · 2026-07-21
- Baseten study finds new facts in LLM weights are fragile unless trained from many restatements — alex_verem · 2026-07-21
- Kimi K3 and Fable 5 now look much closer than the old open-vs-closed gap — FinanceYF5 · 2026-07-21
- uv-scripts/ocr returns to the top of Hugging Face datasets with a JSON model picker — vanstriendaniel · 2026-07-21