Neel Nanda et al.: Decomposing Grokking into Three Phases via Mechanistic Interpretability
burny_tech · x · 2026-10-27
This is the classic interpretability paper (arXiv:2301.05217) by Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt on grokking — where generalization suddenly appears long after training loss has converged.
Key points:
- Using small transformers trained on modular addition, the authors fully reverse-engineer the learned algorithm, which uses discrete Fourier transforms and trig identities to convert addition into rotation around a circle.
- The algorithm is confirmed via activation/weight analysis and ablations in Fourier space.
- Based on this understanding, they define progress measures that split the seemingly discontinuous emergence into three continuous phases: memorization → circuit formation → cleanup.
- Conclusion: grokking is not a sudden shift but the gradual amplification of structured mechanisms in the weights, followed by removal of memorizing components.
More from Research
- Pessimistic complexity theory hypothesis: faster algorithms must get uglier — Elijah_Meeks · 2026-10-27
- UMAP arcing explained: one giant coordinate dominates neural network activation maps — leland_mcinnes · 2026-10-27
- NeurIPS benchmark: tree models still beat deep learning on 45 tabular datasets — tak3sh8 · 2026-10-27
- Kevin Bryan: maximizing raw accuracy is rarely the right way to train AI assistants — Afinetheorem · 2026-10-27
- Szegedy: a flaw in common training losses went unnoticed for over a decade — ChrSzegedy · 2026-10-27
- Claude research model claims to refute Exponential Time Hypothesis with 14M-token autonomous run — scaling01 · 2026-10-27