Anti-grokking: overtraining can collapse generalization, driven by "Correlation Traps"
CatAstro_Piyush · x · 2026-09-26
Researchers identify "anti-grokking": if you keep training past the grokking peak, test accuracy can suddenly collapse back to chance even while training accuracy stays perfect. Standard progress metrics (L2 weight norms, activation sparsity, weight entropy) track the initial grokking phase but stay flat and miss the collapse. Using the WeightWatcher tool to analyze the empirical spectral density of layer weight matrices, the authors show anti-grokking is driven by "Correlation Traps" — anomalously large eigenvalues emerging late in training that severely impair generalization.
More from Research
- Distillation criticized: gradient descent can't even copy the teacher model — deliprao · 2026-09-26
- ResolVI models measurement errors to fix single-cell RNA data, cutting false signals from 14.8% to under 0.01% — bravo_abad · 2026-09-26
- Define Dimensions First: Hamel Husain's Method for Synthetic Eval Data — HamelHusain · 2026-09-26
- LeCun's lab borrows a brain trick to double AI goal-reaching success to 94% — alex_verem · 2026-09-26
- AgentOdyssey, a Text-Game Engine for Test-Time Continual Learning Agents, Accepted at NeurIPS — DanielKhashabi · 2026-09-26
- Boris Cherny's viral tweet puts TLA+ and formal methods in the agent-coding spotlight — fhuszar · 2026-09-26