New paper ties grokking to Singular Learning Theory: generalization as manifold collapse
burny_tech · x · 2026-11-11
burnytech shares the arXiv paper "Grokking From Abstraction to Intelligence" (2603.29262, 22 pages, under review for ICML 2026) by Junjie Zhang et al.
Core claim
- Grokking in modular arithmetic is the classic "fruit fly" experiment for studying the mechanistic origins of generalization, but existing work focuses on local circuits or optimization tuning, overlooking the global structural evolution driving the phenomenon.
- The paper proposes grokking originates from spontaneous simplification of internal model structures governed by the principle of parsimony.
Method & findings
- Integrates causal, spectral and algorithmic complexity measures with Singular Learning Theory (SLT).
- Finds the memorization-to-generalization transition corresponds to physical collapse of redundant manifolds and deep information compression, offering a new lens on overfitting and generalization.
Related event: New research explains grokking as spontaneous structural simplification(2 posts)→
More from Research
- KV cache debate: token identities are just 0.00082% of a 400k context's cache bits — aryaman2020 · 2026-11-11
- LoG 2026 lands in Boston Nov 20-22 with keynotes from Petar Velickovic and more — chaitjo · 2026-11-11
- Must-read papers of the week: HALO humanoid motion, GR00T 2, VLA policies, agent tooling — TheTuringPost · 2026-11-11
- Codex tool-use agent trajectory dataset trends on Hugging Face — israellaguan · 2026-11-11
- Three new math books from Kyoritsu: flow matching for statistical inference, optimal transport, information geometry — FrnkNlsn · 2026-11-11
- EmoHarness: scaffolding lifts small models' multimodal emotion recognition by up to 13.4 points — UIUC-CS · 2026-11-11