Ziming Liu: Popular activations have non-zero mean; negative shift stabilizes training

On October 31, MIT's Ziming Liu (author of KAN and researcher at Moonshot AI) posted on Twitter pointing out a shared "bug" in today's mainstream activation functions ReLU/SiLU/GeLU: these functions are asymmetric, suppressing the negative half-axis more than the positive one, so even with a zero-mean input distribution, the post-activation distribution has a positive mean. He argues this seemingly minor issue can cause pathological behavior in training dynamics, including loss spikes, representation overshooting, an ill-conditioned loss landscape, and slower late-stage convergence.

Confirmed

• In the thread, Ziming Liu gave specific correction constants: ReLU→ReLU−0.2, GeLU→GeLU−0.115, Swish→Swish−0.077, i.e., adding a negative offset to the activation function
• The accompanying experiments show that after the correction, training achieves lower loss in later stages with a smoother loss curve (less oscillation), at the cost of slightly higher loss early on
• He claims the correction improves training stability for both LLMs and ViTs

Why it matters

• The fix is nearly free: just a constant shift, no architectural change; if the finding holds at scale, the payoff is significant
• Activation functions are among the most fundamental building blocks of deep learning; if their non-zero mean indeed causes issues like loss spikes, this discovery will prompt the community to re-examine default design choices

2026-10-31 ~ 2026-10-31 · 8 related posts

• Zero-mean 'bug' in ReLU/SiLU/GeLU found to cause loss spikes; fixing it boosts LLMs and ViTs — ZimingLiu11 · 2026-10-31
• [source] ReLU/SiLU have a zero-mean "bug" — fixing it improves LLM and ViT training — ZimingLiu11 · 2026-10-31
• Non-zero activation mean causes pathological landscapes and slow convergence — ZimingLiu11 · 2026-10-31
• Shifting activation bias slightly (ReLU−0.2, GELU−0.115) lowers late-stage training loss — ZimingLiu11 · 2026-10-31
• [source] Simple activation shift: ReLU-0.2, GeLU-0.115, SwiGLU-style shifts lower late-training loss — ZimingLiu11 · 2026-10-31
• Autoencoder experiments show bias-free activations yield monotonic representation evolution — ZimingLiu11 · 2026-10-31
• Hessian eigenvalue plots show removing activation bias flattens the loss landscape — ZimingLiu11 · 2026-10-31
• [source] Removing activation bias is a free trick that improves ViT, DiT and GPT training — ZimingLiu11 · 2026-10-31