The Depth Delusion: Why Transformers Should Be Wider, Not Deeper
Md Muhtasim Munif Fahim, Md Rezaul Karim
cs.LG, cs.AI
2026-01-29
Across 30 Transformers from 17M to 7B, critical depth scales as W^0.44. At 7B a 32-layer 4096-wide model loses 2.298 nats versus 2.417 for a 64-layer 2816-wide one.
Kaplan and Chinchilla write language-model loss as a power law in parameter count N and token count T, and treat shape as interchangeable. Ten layers at width 8192 and 80 layers at width 2048 should land at the same loss if N matches. Practice still prefers depth: GPT-3 has 96 layers, PaLM 118, on the story that depth buys compositional reasoning.
This University of Rajshahi paper attacks that assumption. It splits loss into a capacity term, a data term, and an architecture penalty that depends only on depth and width, and claims that past a critical depth, extra layers raise loss. They call that the Depth Delusion.
The working picture is exponential gradient decay across layers, with persistence length τ(W)∝W^0.44 (R²=0.98 on the fit). Critical depth Dcrit is where the bottom-layer gradient falls to 1/e of the top, so Dcrit∝W^0.44. In the experimental range they also use 2.43 ln W; at width 512 that gives Dcrit≈15.
Layers beyond Dcrit pay a penalty Φ that grows with the excess-depth ratio and shrinks as width grows. Minimizing under C=6NT yields D∝C^0.12 and W∝C^0.34: the width exponent is about 2.8× the depth exponent. Those 0.12 / 0.34 numbers use Kaplan's α≈0.076, not the α=0.22 this paper fits itself.
They train 30 decoder-only Transformers, depths 2–80, widths 256–6144, from 27M to 7.1B parameters. Pre-LN, RoPE, GELU, no dropout, SlimPajama data. Small models see 6.4B tokens; 1B–7B runs go up to 140B. AdamW at 3×10^{-4}, seed fixed at 42. Compute came from Google TPU Research Cloud.
The global fit is R²=0.922, RMSE=0.113 nats. At fixed width 512, depth is U-shaped:
| Architecture | Params | Loss |
| 16L×512W | 102.1M | 3.435 |
| 24L×512W | 127.3M (+25%) | 3.468 |
| 32L×512W | 152.4M (+50%) | 3.441 |
The 24-layer model has 25% more parameters and 0.033 nats higher loss than 16 layers. 32 layers recover a little versus 24, still lose to 16. At fixed 16 layers, widening from 256 to 1536 drops loss monotonically from 3.929 to 3.049. There is no matching "width delusion."
The same pattern shows up at scale:
| Scale | Better config / loss | Over-deep / loss | ΔL |
| 1B | 24L/1792, 2.821 | 80L/1024, 2.978 | +0.16 |
| 3B | 40L/2432, 2.519 | 72L/1792, 2.681 | +0.16 |
| 7B | 32L/4096, 2.298 | 64L/2816, 2.417 | +0.12 |
At 7B, 32 layers × 4096 (6.86B) land at 2.298; 64 layers × 2816 (6.38B) land at 2.417. The deeper-narrower model has 480M fewer parameters and slightly fewer FLOPs (5.30×10^{21} vs 5.89×10^{21}). That comparison says deep-narrow loses to shallow-wide at similar scale. The clean "more parameters, higher loss" result remains the width-512 table.
Extrapolating with κ=2.43: GPT-3's Dcrit≈22.9 versus 96 layers; PaLM ≈23.9 versus 118; Llama-3 70B ≈21.9 versus 80. Ratios of 3.6–4.9×. That is extrapolation, not a measurement.
If critical depth really scales as W^0.44, extra layers are not free expressivity. Early layers consume parameters and compute while receiving gradients too weak to learn. Mistral 7B's 32 layers happen to match this paper's 7B optimum. For a new pretrain, the heuristic on offer is: stay under D≈2.5 ln W and spend the budget on width.
The claim that challenges Kaplan is "loss depends on shape, not only on N." There is a controlled comparison through 7B. There is none at 100B+.
The fitted capacity exponent α=0.22 has a 95% CI of [-0.21, 0.65], which includes zero, so the optimal-scaling corollary is shaky. Those exponents also borrow Kaplan's α rather than using the paper's own fit.
Every run is a single seed. Small models see only 6.4B tokens, and the 7B pair is not matched on parameters or FLOPs. ReZero, DeepNet, and NormFormer were not tried; the measured Dcrit is for plain Pre-LN and may move if depth is stabilized. The domain is web-text next-token prediction only. Judging GPT-3 and PaLM as 4–5× too deep with 2.43 ln W jumps two orders of magnitude.