Zero-mean 'bug' in ReLU/SiLU/GeLU found to cause loss spikes; fixing it boosts LLMs and ViTs
ZimingLiu11 · x · 2026-10-31
Ziming Liu points out a shared "bug" in common activation functions (ReLU/SiLU/GeLU): post-activation means are not zero. Even tiny, this bias causes training pathologies such as loss spikes and representation overshooting. Removing it improves performance on both LLMs and ViTs.
More from Research
- Overfitting explained with two students and two exams — cneuralnetwork · 2026-10-31
- Betelgeuse runs 68GB Qwen MoE bit-exact at 22 tok/s on RTX 3060 + 16GB RAM — zyxciss · 2026-10-31
- EdgeRock-4B open-sourced with fixed 72MiB KV cache, 99.1% less memory than base — testingcatalog · 2026-10-31
- Claude's Muon optimizer theorem checks out numerically to 4 decimal places — vxnuaj · 2026-10-31
- Dev asks for a mathematician: Claude's Muon optimizer theory is too deep to verify — vxnuaj · 2026-10-31
- Biology Isn't Just a Data Problem: The lncRNA ANRIL Breaks Every Model You'd Write — iskander · 2026-10-31