Is recurrent depth the real driver? Researchers debate 3x depth vs 81x params math
xuanalogue · x · 2026-09-04
Ryan Greenblatt argues that recent gains in opaque reasoning ability are likely downstream of architecture changes, not parameter scaling. Parsing Jakub's claim of being "within a factor of 2 of GPT-4", he estimates a 3x increase in depth — which, based on open-weight scaling trends, would normally require 81x more parameters.
@xuanalogue questions the math: by default a 3x depth increase should be roughly equivalent to 3x parameters, and looped transformers actually yield less due to parameter reuse and lack of weight specialization — unless added depth enables other scaling increases.
More from Research
- Richard Socher: AI could compress a century of scientific breakthroughs into a decade — RichardSocher · 2026-10-21
- Startup Electric Plant builds an AI model to decipher plants' electrical signals — TechCrunch AI · 2026-10-21
- Eval researcher teases guide to subjective-task evals, with a curated resource list — cwolferesearch · 2026-10-20
- IIT Madras article splits LLM memory into five approaches and asks why belief revision is still missing — LimeOptimal539 · 2026-10-20
- Scholar writes AI-assisted book on storytelling values, work nominated for EMNLP best paper — _akpiper · 2026-10-20
- Google's T2J-Bench shows coding agents score just 0.676 migrating PyTorch repos to JAX — CShorten30 · 2026-10-20