Scaling depth 3x means 81x params: Ryan Greenblatt on looped transformer scaling
RyanGreenblatt · x · 2026-09-04
In an X exchange on looped transformers, Ryan Greenblatt notes that scaling the architecture "normally" means each 3x increase in depth yields an 81x increase in parameters (likely active params, though details are fuzzy) — challenging the naive assumption that depth is a cheap axis to scale.
More from Research
- Stanford team launches ScholarCatalyst benchmark for inspiration retrieval — lateinteraction · 2026-10-21
- Three reasons AI is not conscious and may never be, per Nature essay — anilkseth · 2026-10-21
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- SFT details: instruction/reasoning/code/math mix, synthetic + human traces, no epoching — TimDarcet · 2026-10-21
- Mid-training recipe disclosed: 32k→131k context extension, dropping logit distillation for tokens — TimDarcet · 2026-10-21
- Mathematician Daniel Litt concedes his AI bet, but says AI won't solve math — ziv_ravid · 2026-10-21