New grokking research explains plateau length, weight decay and init effects
CatAstro_Piyush · x · 2026-10-21
Researcher dohmatobelvis and PhD student gningbayemalick released new work offering deep insights into the still poorly understood grokking phenomenon.
The work addresses core questions:
- What controls the length of the plateau
- Whether weight decay is necessary for grokking
- The impact of initialization
The accompanying figure shows a fast/slow dynamics decomposition: learning relaxes quickly onto an invariant manifold, a slower phase at timescale 1/ε (set by SNR) learns the core feature weight and accounts for the long plateau, and a final slowest phase—present only with unbalanced initialization—relaxes the "harmonic dual" variable at timescale 1/λ set by weight decay.
More from Research
- EvoMusArt 2027 opens submissions, due Nov 1, 2026 in Mainz, Germany — evomusart_conference · 2026-10-21
- AI denoising recovers superconductor spectra in 12s vs 5h, with error bars on the result — bravo_abad · 2026-10-21
- LLM2R recasts 3D part assembly as sequence modeling with a decoder-only LLM — NationalUniversityofSingapore · 2026-10-21
- Ant International's FlowState treats execution state as memory for long-horizon LLM agents — ant-intl · 2026-10-21
- Google's TEAL evolves reusable analytics tools for coding agents without ground-truth labels — google · 2026-10-21
- Tencent's ASAP evaluates agentic capabilities in pretrained base models before post-training — tencent · 2026-10-21