Common activation functions share a nonzero-mean bug that causes loss spikes
samiramanabi · x · 2026-11-02
Ziming Liu (MIT) points out that widely used activation functions (ReLU/SiLU/GeLU etc.) share a "bug": the mean of post-activations is not zero. Though tiny, this leads to pathologies in training dynamics, including loss spikes and representation overshooting. Removing the bug improves performance in both LLMs and ViTs. Reposters quip that there may be no part of the deep learning algorithm stack that isn't riddled with bugs.
More from Research
- Auditor says viral weight-editing paper overhyped but safety concerns are real — RichmanRonald · 2026-11-04
- Tom Dietterich: Unlearning methods are a form of weight space programming — tdietterich · 2026-11-04
- New task formalizes semantic anomaly detection: spotting commonsense errors in AI-generated images — burkov · 2026-11-04
- Berkeley's Value Forcing: enforcing Bellman consistency at every noise level for accurate RL value functions — berkeley_ai · 2026-11-04
- Living review protocol registered on OSF, MIT-licensed code, PRs welcome — maier_ak · 2026-11-04
- A systematic review that keeps itself up to date, powered by LLMs — maier_ak · 2026-11-04