ReLU/SiLU have a zero-mean "bug" — fixing it improves LLM and ViT training
ZimingLiu11 · x · 2026-10-31
Ziming Liu (MIT) argues that common asymmetric activation functions (ReLU/SiLU/GeLU) suppress the negative side more than the positive, so post-activation distributions have positive mean even with zero-centered inputs. This small "bug" causes pathological loss landscapes, loss spikes, and representation overshooting; switching to shifted, symmetric activations improves training in both LLMs and ViTs.
More from Models
- Anthropic's fable 5.5 reportedly delayed, possibly overshadowed by Opus 5.5 — haider1 · 2026-10-31
- Anthropic's Max→Ultra rename checked line by line: weekly caps never went away — No-News-8059 · 2026-10-31
- Google Insiders Push Back on 'No Great Models' Criticism, Carbon Is Just Benchmarks — Angaisb_ · 2026-10-31
- OpenAI launches GPT-6.1 Terra: 50.3% on Terminal-Bench 4.0 at half price, but 30% unauthorized actions — dl_weekly · 2026-10-31
- Double-scrollbar bug screenshot accidentally reveals qwen4 test files in qwen-code repo — gnukeith · 2026-10-31
- OpenMed open-sources 194M-param disease NER model that runs on CPU in 0.21s — solyarisoftware · 2026-10-31