Researchers argue computational depth is the missing scaling axis, stuck at ~100 layers since GPT-3
burny_tech · x · 2026-09-24
ethantsliu and Akshay Vegesna argue that computational depth is the missing scaling axis: params, data, sparsity, and test-time reasoning have all been scaled by orders of magnitude, but depth has been stuck at 100 layers since GPT-3.
Key claims:
- Forcing a 100,000-bit hidden state into 17-bit text tokens for chain-of-thought is a massive bottleneck and a wasteful human prior
- LLMs are both severely depth-bottlenecked and bad at using the depth they have
- Architectural interventions that efficiently lift this bottleneck yield gains that grow with compute
This is framed as scaling computational depth per the bitter lesson; full paper details are not yet provided.
Related event: Q Labs: Computational Depth Is the Missing Scaling Axis for LLMs(4 posts)→
More from Research
- Ross Girshick posts CVPR slides recounting 1.5 decades of object detection and its link to the LLM era — CatAstro_Piyush · 2026-11-11
- KAIST's WikiTrace: evidence-linked memory fixes agent "memory-induced degradation" — dair_ai · 2026-11-11
- UHN researchers unveil DynaMELD, an ML algorithm to prioritize liver transplant waitlists more equitably — VectorInst · 2026-11-11
- What it takes to squeeze performance: EMA, cosine scheduling and custom dead-cluster revival — chrisemoody · 2026-11-11
- Trainable JWST model boosts exoplanet sensitivity 2–15x in Nature paper — bravo_abad · 2026-11-11
- Ben Todd's alt log-log plot shows task horizons hitting a singularity in under 12 months — ben_j_todd · 2026-11-11