KAN author Ziming Liu's new paper: loss spikes are predictable, quadratic loss picture overturned
KAN author and MIT postdoc Ziming Liu released a new paper on day five of the MetaCircle AI Research showcase, tackling the long-standing problem of loss spikes in LLM pretraining. The core finding: loss spikes are predictable. The team mapped a phase diagram over optimizer (β₁, β₂) parameters showing a stable-vs-spiky boundary, enabling practitioners to foresee and avoid spiky configurations in real training runs. This overturns the prevailing quadratic picture of the training loss, and the team's physics-style analysis further suggests beta1=beta2 may be optimal. Liu shared the paper link in his thread.
Confirmed
- Spiky behavior is just one phase in the (β₁, β₂) optimizer parameter plane, with a stable-spike boundary whose measured slope is close to 1, allowing prediction of which configurations produce spikes.
- The physics-style analysis suggests beta1=beta2 may be optimal; the quadratic model severely mischaracterizes training spikes.
- Spikes are not unique to LLM pretraining: they also appear in simpler systems such as learning modular arithmetic.
- Explaining why spikes occur only mid-training, Liu notes that in early and late stages the effective LR never catches up with the critical LR; only the middle stage spikes, making learning rate the key factor.
- As an AI-for-science demonstration, the team studied how the exponent n in loss(x)=|x|^n changes Adam's stability phase diagram, finding n=2 and n=2.1 yield drastically different diagrams—optimizers are highly sensitive to the loss form.
- Liu cited the slingshot mechanism paper linking training spikes to grokking, and further proposed that training loss spikes may trigger grokking.
- One explanation for spike causes is rare tokens: the model behaves abnormally on rarely seen samples, producing loss spikes.
Why it matters
- Loss spikes are a persistent pain in large-scale pretraining; predicting and avoiding them via the optimizer parameter plane could directly reduce run interruptions and wasted compute.
- Linking spikes to effective/critical learning rates, grokking, and the slingshot mechanism offers a unified view of training dynamics and challenges the quadratic loss landscape assumption.
- The controlled loss(x)=|x|^n experiments showcase an AI-for-science approach—using minimal systems to reveal mechanisms of complex phenomena—with value for optimizer selection and training stability research.
2026-11-01 ~ 2026-11-01 · 10 related posts
- Episode 1: Ziming Liu flags non-zero mean bug in ReLU/SiLU/GeLU; a constant shift fixes it(2026-10-31, 9 posts)
- Episode 2: KAN author Ziming Liu's new paper: loss spikes are predictable, quadratic loss picture overturned(2026-11-01, 10 posts)
Primary sources
- KAN author's new paper: loss spikes are predictable, quadratic loss picture wrong — ZimingLiu11 · 2026-11-01
- [source] Ziming Liu shares loss spike paper link, MetaCircle exposition Day 5 — ZimingLiu11 · 2026-11-01
- Rare tokens may be the culprit behind loss spikes in LLM training — ZimingLiu11 · 2026-11-01
- Ziming Liu: training loss spikes may trigger grokking via the slingshot mechanism — ZimingLiu11 · 2026-11-01
- Ziming Liu maps momentum-driven loss spike phase diagram in modular arithmetic training — ZimingLiu11 · 2026-11-01
- Adam phase diagrams shift drastically from loss power n=2 to n=2.1, study finds — ZimingLiu11 · 2026-11-01
- [source] Adam stability phase boundaries have slope near 1, not 3: loss landscapes aren't quadratic — ZimingLiu11 · 2026-11-01
- Ziming Liu explains loss spikes: only the middle phase spikes as effective LR crosses critical LR — ZimingLiu11 · 2026-11-01
- [source] Ziming Liu maps loss-spike phase diagram, shows quadratic model misreads training spikes — ZimingLiu11 · 2026-11-01
- Physics-style analysis maps loss spike phase diagram, hints beta1=beta2 optimal — ZimingLiu11 · 2026-11-01