New paper finds 'anti-grokking': test accuracy collapses back to chance after successful generalization
sytelus · x · 2026-09-20
A new arXiv paper by Hari Prakash and Charles Martin extends two canonical grokking experiments (a 3-layer MLP on MNIST subset and a transformer on modular addition) far beyond standard training and reports a previously unreported third phase: anti-grokking — after transitioning to successful generalization, test accuracy collapses back to chance while training accuracy stays perfect.
- Diagnosis uses the open-source WeightWatcher tool (HTSR/SETOL theory). The primary signal is Correlation Traps: anomalously large eigenvalues beyond the Marchenko–Pastur bulk in the spectral density of shuffled weight matrices; a secondary signal is the average HTSR layer quality metric α deviating from 2.0.
- Neither metric requires access to train or test data — overfitting signatures are read directly from weight matrices.
- The authors argue this late-stage overfitting is qualitatively different from previously observed forms.
Related event: Paper Reveals Third Stage of Grokking: Late-Stage Generalization Collapse(2 posts)→
More from Research
- Meta's Atomic Creator Tokens beat Semantic IDs: +19.8pt Recall@15 and 2.7x lower latency — _reachsumit · 2026-11-04
- HiGenTrans transfers LLM knowledge into generative recommenders at input, representation and decision levels — _reachsumit · 2026-11-04
- SA-OPSD distills self-teacher knowledge at beam-search pruning boundaries for generative recommendation — _reachsumit · 2026-11-04
- MV-Thinker: Multi-View Thinking with Adaptive Generation for Generative Recommendation — _reachsumit · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Kuaishou's UAGR injects utility signals into semantic codes and decoding for generative recommendation — _reachsumit · 2026-11-04