Depth-growing 3B models beat dense training on GSM8K (19% vs 12%) while saving 26% pretraining compute
burny_tech · x · 2026-11-01
A paper from TUM, Helmholtz AI, MCML and Google's Paradigms of Intelligence team examines how depth growing (progressively duplicating layer blocks during training) and looping (reusing a shared block) shape in-context learning (ICL).
Key findings:
- At 3B scale, progressively duplicating middle layers beats standard dense training on 8-shot GSM8K (19% vs 12% exact match) while cutting pretraining compute by 26%.
- Grown and looped models rely more on in-context information and show ICL earlier on controlled tasks, with learned associations staying robust in the weights.
- Attention-head tracing shows depth growth propagates induction heads across layers early in training; ablations show grown/looped models lean more strongly on induction heads for few-shot tasks and later-token prediction.
- Conclusion: computation reuse creates a shared inductive bias toward learning from context (soft parameter reuse), shaping how ICL develops.
More from Infra
- RAM up 7x to $429 in 18 months as Sam Altman says the invisible hand is failing — aakashgupta · 2026-11-01
- Blogger estimates xAI on track for $100B ARR by end-2026 as datacenter builds move up — mitchdeg · 2026-11-01
- From-scratch CUDA engine runs full GLM-5.3 on 3-6 DGX Sparks; steering vectors flip output language — burny_tech · 2026-11-01
- How NVIDIA's instruction scheduling model works: static latencies and scoreboards — mike64_t · 2026-11-01
- Picked up a $1,100 RTX 3090 FTW3 for a headless local LLM server — seeking configs — NighthawkXL · 2026-11-01
- Sequoia's Shaun Maguire builds SpaceX model projecting $731B revenue by 2028 — FinanceYF5 · 2026-11-01