Ziming Liu maps momentum-driven loss spike phase diagram in modular arithmetic training
ZimingLiu11 · x · 2026-11-01
Ziming Liu continues his thread on training spikes: they appear not only in LLM training but also in simple systems like learning modular arithmetic, which—per the slingshot mechanism paper—links spikes to grokking. He uses it as a controllable model organism.
The new finding: the momentum factors beta1 and beta2 strongly shape spiky behavior. He presents a phase diagram for modular arithmetic training and builds a toy model from quadratic loss L(x)=x². The two are qualitatively similar, but the phase boundary slopes differ sharply—about (1-β₂)∝(1-β₁)^0.92 for modular arithmetic vs 3.07 for the toy model, showing the simple picture doesn't fully capture real spiking dynamics.
More from Research
- Paperclip builds biomedical datasets in one shot, cutting months-long curation to about a minute — james_y_zou · 2026-11-04
- Embodied Agent Arena: Closed-Weight VLMs Lead All Five Robot Task Domains, Open-Weight Models Lag — Haojian Huang · 2026-11-04
- FPT University Paper Shows RAG Improves Stock Market Information Retrieval Accuracy and Latency — Huan Phong Thai · 2026-11-04
- Alibaba's adaptive GUI agent training cuts environment interactions by 56-61% — Yuxiang Chai · 2026-11-04
- Large-Scale Study: AI Reviewers Often Ignore How Much You Improved Your Paper — umd-zhou-lab · 2026-11-04
- Kling Team's report: video generation bottleneck shifts from rendering to semantic control — KlingTeam · 2026-11-04