FULL STORY

KAN Author's New Work: Activation Flaws and Stable Training

KAN author Ziming Liu's new paper identifies non-zero mean activations like ReLU/SiLU/GeLU as a bug, shows negative bias stabilizes training, and makes loss spikes predictable via phase diagrams.

2026-10-31 ~ 2026-11-02 · 2 episodes · 19 posts

Episode 1 · Ziming Liu flags non-zero mean bug in ReLU/SiLU/GeLU; a constant shift fixes it (2026-10-31, 9 posts)

On Oct 31, MIT's Ziming Liu (KAN author, Moonshot AI researcher) posted a thread arguing that mainstream activations ReLU/SiLU/GeLU share a common "bug": they are asymmetric, suppressing the negative half more than the positive, so even with zero-mean inputs the post-activation mean is positive. He links this to pathological training dynamics including loss spikes, representation overshooting, pathological loss landscapes and slower late-stage convergence; the fix is adding a negative shift to the activation.

Confirmed

  • Fix constants: ReLU→ReLU−0.2, GeLU→GeLU−0.115, Swish→Swish−0.077 — a zero-cost constant shift requiring no architectural change
  • Experiments show lower late-stage loss and smoother (less oscillatory) loss curves after the fix, at the cost of slightly higher early loss
  • Hessian eigenvalue experiments (Step 0/5000/10000) show top eigenvalues dropping from roughly 20–50 to around 2 after removing activation bias, indicating a consistently flatter loss landscape
  • The author reports gains across ViT (image classification), DiT (image generation) and GPT (language pretraining), with improved training stability
  • Autoencoder experiments: models with activation bias can recover identifiable image content from the bias component early in training, but this dependence fades; without bias, representation evolution is monotonic and the bias component carries almost no image content

Why it matters

  • The fix is nearly free — a single constant shift — so if it holds at scale, the payoff is significant
  • Activations are among the most fundamental deep learning building blocks; if non-zero means truly cause loss spikes and pathological optimization, the community may need to rethink default design choices

Episode 2 · KAN author Ziming Liu's new paper: loss spikes are predictable, quadratic loss picture overturned (2026-11-01, 10 posts)

KAN author and MIT postdoc Ziming Liu released a new paper on day five of the MetaCircle AI Research showcase, tackling the long-standing problem of loss spikes in LLM pretraining. The core finding: loss spikes are predictable. The team mapped a phase diagram over optimizer (β₁, β₂) parameters showing a stable-vs-spiky boundary, enabling practitioners to foresee and avoid spiky configurations in real training runs. This overturns the prevailing quadratic picture of the training loss, and the team's physics-style analysis further suggests beta1=beta2 may be optimal. Liu shared the paper link in his thread.

Confirmed

  • Spiky behavior is just one phase in the (β₁, β₂) optimizer parameter plane, with a stable-spike boundary whose measured slope is close to 1, allowing prediction of which configurations produce spikes.
  • The physics-style analysis suggests beta1=beta2 may be optimal; the quadratic model severely mischaracterizes training spikes.
  • Spikes are not unique to LLM pretraining: they also appear in simpler systems such as learning modular arithmetic.
  • Explaining why spikes occur only mid-training, Liu notes that in early and late stages the effective LR never catches up with the critical LR; only the middle stage spikes, making learning rate the key factor.
  • As an AI-for-science demonstration, the team studied how the exponent n in loss(x)=|x|^n changes Adam's stability phase diagram, finding n=2 and n=2.1 yield drastically different diagrams—optimizers are highly sensitive to the loss form.
  • Liu cited the slingshot mechanism paper linking training spikes to grokking, and further proposed that training loss spikes may trigger grokking.
  • One explanation for spike causes is rare tokens: the model behaves abnormally on rarely seen samples, producing loss spikes.

Why it matters

  • Loss spikes are a persistent pain in large-scale pretraining; predicting and avoiding them via the optimizer parameter plane could directly reduce run interruptions and wasted compute.
  • Linking spikes to effective/critical learning rates, grokking, and the slingshot mechanism offers a unified view of training dynamics and challenges the quadratic loss landscape assumption.
  • The controlled loss(x)=|x|^n experiments showcase an AI-for-science approach—using minimal systems to reveal mechanisms of complex phenomena—with value for optimizer selection and training stability research.