Depth-Grown and Looped LMs Learn In-Context Learning Earlier Than Dense Models
omarsar0 · x · 2026-10-26
A recommended paper (TUM/Helmholtz AI/Google) examines how layer reuse in looped and depth-grown language models shapes in-context learning.
- At 3B parameters, both grown and looped models gain more from later context than a same-depth dense model, scoring higher on GSM8K at every number of demonstrations and predicting unseen formal languages more accurately.
- In controlled tasks they acquire ICL earlier: when a task can be solved from context or by memorizing fixed labels in weights, they keep relying on context.
- Mechanistically, growing depth by copying layers duplicates early induction heads, and most strongest final heads descend from a few early ancestors; looped models repeat the same pattern across passes.
Takeaway: layer reuse confers a shared inductive bias toward learning from context, and soft parameter reuse shapes how ICL develops.
More from Models
- Grok Bot hands-on: auto-logs hundreds of tasks to Linear and runs scheduled security checks — koltregaskes · 2026-10-26
- Muse Spark 1.3 hits 0.6% attack success rate on Skill-Inject prompt injections — maksym_andr · 2026-10-26
- Claim: Opus 5.5 is the best model ever released, with RSI taking off — iruletheworldmo · 2026-10-26
- Anthropic paper finds functional emotion representations inside Claude Sonnet 4.5 — mervenoyann · 2026-10-26
- Grok Bot teased as a voice agent that handles customer service calls and refunds for you — XFreeze · 2026-10-26
- Gemini 4 Argon lands #3 on RSI-Exam self-improvement leaderboard — HuaxiuYaoML · 2026-10-26