Implicit Curriculum Hypothesis: LLM Skills Emerge in Predictable Order, CMU Paper Finds
CatAstro_Piyush · x · 2026-10-17
CMU researchers (Emmy Liu, Graham Neubig, et al.) propose the Implicit Curriculum Hypothesis: capability emergence during LLM pretraining follows a compositional, predictable order across models and data mixtures.
- They built a suite of simple composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and math, tracking emergence points across 4 model families from 410M–13B parameters.
- Emergence orderings at fixed accuracy thresholds are strikingly consistent (ρ=.81 across 45 model pairs), and composite tasks almost always emerge after their components.
- This structure is encoded in representations: tasks with similar function vectors follow similar training trajectories.
- Using the representation space, they can predict training trajectories of held-out compositional tasks throughout pretraining (R²=.68–.84) without access to training data.
More from Research
- "Just add more RL envs": the one-line mantra of the RL scaling era — burny_tech · 2026-10-17
- JevBench Adds 34 Decision Models in 4 Days as kettei-fat Takes #1 — airesearch12 · 2026-10-17
- Diffusion Gemma promises 4x faster text generation; may supplant autoregressive LLMs — NandoDF · 2026-10-17
- Kyutai open-sources REKI, a 2.4B VLA trained on public data only, setting a new size-performance Pareto frontier — MikePFrank · 2026-10-17
- Most rerankers hurt retrieval: gains only show up at 4B scale, per new testing — bo_wangbo · 2026-10-17
- Mathematicians dispute OpenAI's Navier-Stokes proof: it may not understand its own 166-page solution — The Decoder · 2026-10-17