Paper: strategy coopetition explains why in-context learning emerges and then disappears
scychan_brains · x · 2026-09-01
arXiv 2503.05631, from authors at DeepMind and Oxford (Aaditya K. Singh, Felix Hill, Stephanie C.Y. Chan, et al.), studies why in-context learning (ICL) emerges during training and later disappears.
Key findings:
- After ICL vanishes, the model's asymptotic strategy is a hybrid of in-weights and in-context learning the authors call CIWL (context-constrained in-weights learning);
- CIWL and ICL both compete and share sub-circuits — "strategy coopetition": ICL cannot emerge quickly on its own and is enabled by the slow development of CIWL;
- They propose a minimal mathematical model reproducing these dynamics and use it to identify a setup where ICL is truly emergent and persistent.
More from Research
- Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
- semiconductor.diy: open-source guides to make real chips in a garden shed — jwt0625 · 2026-10-18
- Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
- Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs — inductionheads · 2026-10-18
- q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18
- Smarter Every Day's Destin to defend PhD dissertation on impact fusion — CatAstro_Piyush · 2026-10-18