Researchers argue computational depth is the missing scaling axis, stuck at ~100 layers since GPT-3

burny_tech · x · 2026-09-24

ethantsliu and Akshay Vegesna argue that computational depth is the missing scaling axis: params, data, sparsity, and test-time reasoning have all been scaled by orders of magnitude, but depth has been stuck at 100 layers since GPT-3.

Key claims:

Forcing a 100,000-bit hidden state into 17-bit text tokens for chain-of-thought is a massive bottleneck and a wasteful human prior
LLMs are both severely depth-bottlenecked and bad at using the depth they have
Architectural interventions that efficiently lift this bottleneck yield gains that grow with compute

This is framed as scaling computational depth per the bitter lesson; full paper details are not yet provided.

Original post →

Researchers argue computational depth is the missing scaling axis, stuck at ~100 layers since GPT-3

More from Research