Scaling depth 3x means 81x params: Ryan Greenblatt on looped transformer scaling
RyanGreenblatt · x · 2026-09-04
In an X exchange on looped transformers, Ryan Greenblatt notes that scaling the architecture "normally" means each 3x increase in depth yields an 81x increase in parameters (likely active params, though details are fuzzy) — challenging the naive assumption that depth is a cheap axis to scale.
Related event: Greenblatt: 3x Depth Gain Comes From Architecture, Not Parameter Scaling(4 posts)→
More from Research
- AI BioDesign accelerator launches with UW Medicine and Fred Hutch to let AI design biology — AllThingsApx · 2026-09-04
- Sarvam AI releases Vāgartha: 217,959 Sanskrit verses with detailed explanations on Hugging Face — selfawareatom · 2026-09-04
- ORNL's AI system arranges molecules atom-by-atom, working 25+ hours unattended — Scobleizer · 2026-09-04
- OpenAI researcher: GPT-6's CoT controllability keeps rising over RL training — gleech · 2026-09-04
- Percolation Dynamics in Optimization: Variance Cascades in SGD, Adam and AdamW — Sai Niranjan Ramachandran · 2026-09-04
- Compile by Training: Distilling Natural-Language Specs into Local Neural Functions — UWaterloo · 2026-09-04