New grokking study: generalization emerges from spontaneous structural collapse and compression
burny_tech · x · 2026-11-11
A new grokking paper proposes that the memorization-to-generalization transition in modular arithmetic stems from a spontaneous simplification of internal model structures governed by parsimony. Combining causal, spectral, and algorithmic complexity (BDM) measures with Singular Learning Theory, the authors show: representation space collapses from a high-entropy state into a low-dimensional 1D ring manifold isomorphic to Z97; Gini coefficient and inverse participation ratio spike sharply as embedding energy localizes into sparse Fourier modes; and weight algorithmic complexity drops abruptly with parameters becoming block-structured, signaling physical collapse of redundant manifolds and deep information compression. Framed as a new lens on overfitting and generalization mechanisms.
Related event: New research explains grokking as spontaneous structural simplification(2 posts)→
More from Research
- KV cache debate: token identities are just 0.00082% of a 400k context's cache bits — aryaman2020 · 2026-11-11
- LoG 2026 lands in Boston Nov 20-22 with keynotes from Petar Velickovic and more — chaitjo · 2026-11-11
- Must-read papers of the week: HALO humanoid motion, GR00T 2, VLA policies, agent tooling — TheTuringPost · 2026-11-11
- Codex tool-use agent trajectory dataset trends on Hugging Face — israellaguan · 2026-11-11
- Three new math books from Kyoritsu: flow matching for statistical inference, optimal transport, information geometry — FrnkNlsn · 2026-11-11
- EmoHarness: scaffolding lifts small models' multimodal emotion recognition by up to 13.4 points — UIUC-CS · 2026-11-11