Ziming Liu: training loss spikes may trigger grokking via the slingshot mechanism
ZimingLiu11 · x · 2026-11-01
KAN author Ziming Liu continues his thread on LLM training loss spikes: spikes appear even in simple systems like modular arithmetic learning, and the slingshot mechanism paper argues spikes lead to grokking. The attached figure shows the cyclic pattern—last-layer weight growth, loss spike, then norm plateau—in a ReLU FCN. The team picks this simple, controllable setting as a model organism for studying grokking.
More from Research
- Paperclip builds biomedical datasets in one shot, cutting months-long curation to about a minute — james_y_zou · 2026-11-04
- Embodied Agent Arena: Closed-Weight VLMs Lead All Five Robot Task Domains, Open-Weight Models Lag — Haojian Huang · 2026-11-04
- FPT University Paper Shows RAG Improves Stock Market Information Retrieval Accuracy and Latency — Huan Phong Thai · 2026-11-04
- Alibaba's adaptive GUI agent training cuts environment interactions by 56-61% — Yuxiang Chai · 2026-11-04
- Large-Scale Study: AI Reviewers Often Ignore How Much You Improved Your Paper — umd-zhou-lab · 2026-11-04
- Kling Team's report: video generation bottleneck shifts from rendering to semantic control — KlingTeam · 2026-11-04