DepthBench: Measuring How Residual Connections Enable More Computational Depth
Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu, Shiwei Liu
cs.CV, cs.AI, cs.LG
2026-09-26
DepthBench sweeps 10 residual designs at ~400M: Pre-LN val loss rises from 2.759 (16L) to 2.782 (32L); HC falls from 2.729 to 2.699, and Full AttnRes also improves with depth.
Making a Transformer deeper is supposed to buy a longer chain of nonlinear maps. Pre-LN residual connections make that chain trainable. They do not make later layers do useful work. Updates get weaker, more redundant, and more similar to the layers above them, so extra layers raise nominal depth without a matching rise in effective computation. That failure mode is the curse of depth: architectural depth grows, computational depth does not.
LNS and KEEL retune normalization. Hyper-Connections (HC) expand the residual into several learned streams; AttnRes mixes earlier layer outputs with attention. Both families, plus mHC, report better optimization and quality at scale, and related designs already show up in Kimi K3 and DeepSeek V4. Those papers were not run on the same model shapes, recipes, or system budgets, so the gains cannot be pinned on depth actually being used. DepthBench asks a narrower question: under a fixed parameter budget, which designs turn extra layers into extra sequential computation, so that depth is a real scaling axis.
The benchmark treats depth as a capacity split. Parameter count and the pre-training recipe stay roughly fixed, while the aspect ratio dmodel / nlayer moves from shallow-wide to deep-narrow. Depth is then not confounded with a bigger model.
All variants share a LLaMA-like backbone. Sub-1B models use 16-head MHA, RoPE, RMSNorm, and SwiGLU. The 1.6B suite switches to the Qwen3-1.7B GQA layout (16 query heads, 8 KV heads). Pre-training is from scratch on FineWeb-Edu with a Chinchilla-style 20 tokens per parameter (8B tokens at 400M, 32B at 1.6B), sequence length 2048, and a global batch of about 1M tokens per step. Each architecture gets its own learning-rate sweep; every shape is reported at that architecture's best LR.
Ten designs fall into four groups:
The main suite is about 400M parameters, seven shapes, aspect ratios from 76.0 (16 layers × 1216) to 9.1 (70 layers × 640). Three controls sit around it: a 300M iso-backbone sweep that holds embedding share constant, a 200M to 500M multi-scale check, and three shapes at 1.6B. Layer tests use angular distance, a causal score (how much skipping layer s changes a later update), and a permutation score (how much swapping two layers raises LM loss). Aggregation uses only the last three quarters of the network, because the first layers are important in almost every design and would swamp a global mean.
On the 400M sweep, Pre-LN and most norm variants do not like going deeper. Their optima sit at aspect ratios 42.7 to 76.0. HC and Full AttnRes keep improving into the extreme deep-narrow corner.
| architecture | val loss at 16 layers | val loss at 32 layers | depth trend |
| Pre-LN | 2.759 | 2.782 | worse when deeper |
| Full AttnRes | 2.751 | 2.718 | better when deeper |
| HC | 2.729 | 2.699 | better when deeper |
| Block AttnRes | best 2.715 at 20 layers | 2.730 | peak in the middle |
The HC and Full AttnRes trend survives both a fixed total size and a fixed backbone size, out to 70 layers at aspect ratio 9.1. With backbone size held fixed, deep-narrow models can have a smaller total parameter count and still drop loss, so the gain is the width-depth split, not a bigger backbone. The derived variants do not inherit this: mHC peaks at 24 layers and has no consistent depth gain.
Teacher-forced NLL on coding (MBPP, HumanEval), STEM (SciQ, GPQA), and math (GSM8K, MATH-500) follows the same split. Deeper HC and Full AttnRes get lower NLL on STEM and math; Pre-LN and most others are weak or non-monotonic. The paper does not report generative accuracy on those tasks.
Layer diagnostics line up with the loss curves. For L=32, the share of late-layer pairs with causal score above 0.45 is about 1% for Pre-LN, 16% for Full AttnRes, and 7% for HC. The share with permutation score above 0.15 is about 8%, 46%, and 35%. Pre-LN hidden states become angularly similar with depth. HC and Full AttnRes keep non-smooth, layer-specific changes. They do it differently. Full AttnRes retrieves stored earlier outputs. HC mixes several residual streams.
Block AttnRes has a dead prefix: following the Kimi K3 schedule of 8 block boundaries, block size grows with depth, and at L=32 the first 8 layers put almost no weight on their own computation. That is a quarter of the network. mHC residual products have effective rank 1.44 to 1.65 against 2.53 to 2.81 for HC; the doubly stochastic constraint collapses streams toward uniform mixing and cuts long-range route diversity.
Depth is not free. On 400M BF16 prefill, sequence 4096, A100, shrinking the aspect ratio from 76.0 to 9.1 raises FLOPs from about 3.0 to 4.4 TFLOPs per sequence and more than doubles KV cache. Full AttnRes has the highest peak memory. HC has the highest GPU-hour cost.
At 1.6B, Full AttnRes still prefers deeper shapes. HC hits gradient explosion on the widest 500M run, and the 1.6B depth trend is no longer clean. HC is more sensitive to optimization hyperparameters, which is why mHC exists.
This measurement paper treats residual wiring as a scaling-axis question. On Pre-LN, going deep and narrow at fixed parameter count usually hurts. If depth is meant to be a scaling axis, the residual topology has to change; the two designs with a clear signal here are HC and Full AttnRes. Norm patches (LNS, Sandwich-LN, DeepNorm, KEEL) barely move the iso-parameter scan.
That also explains why models such as Kimi K3 and DeepSeek V4 started touching residuals, and what the cheaper cousins give up. mHC is more stable and cuts away HC's depth gains. Block AttnRes is cheaper and idles the early layers under a fixed-boundary schedule. Lower loss comes with more prefill, more KV, higher peak memory, and more GPU time.
The scale is still modest. The 400M picture is clean; 1.6B only has three shapes, and HC is already shaky. Treat this as a residual-design scorecard, not a blueprint that deep-narrow always wins.
The paper is explicit about the systems cost and about HC's learning-rate sensitivity at larger scale. The 1.6B runs skip the per-architecture LR sweep used at 400M and all use 5e-4, so how much of HC's weaker trend is under-tuning is unknown.
Domain evaluation stops at teacher-forced NLL. There is no HumanEval pass@k or GSM8K accuracy, and lower NLL is not the same as solving the task. Causal and permutation thresholds (0.45 and 0.15) are chosen; the heatmaps support the qualitative claim, but there is no threshold-sensitivity study. The main recipe is 400M, FineWeb-Edu, context 2048. In long context the T²Ld attention term gets heavier, and the systems penalty of depth would grow. The paper measures 4096 prefill and counts FLOPs; it does not train long-context models. MoDA is grouped with cross-layer access, yet behaves like Pre-LN, and the paper does not explain why.