Why language models moved from ReLU to GELU, explained
cneuralnetwork · x · 2026-11-11
A concise technical explainer on why modern language models use GELU instead of ReLU:
- ReLU (max(0, x)) zeroes out negative inputs; GELU (x·Φ(x)) smoothly weights inputs by magnitude using the standard normal CDF — e.g. GELU(-1) ≈ -0.159, letting some negative signal through instead of a hard cutoff.
- Introduced by Hendrycks & Gimpel in 2016, it gained traction after adoption in BERT and GPT-style models.
- The author notes GELU isn't universally better than ReLU: its popularity rests on empirical results, not a theorem proving optimality.
More from Research
- Review of 6 RSI benchmarks finds flawed tasks in 5, with 31-63% error rates — teortaxesTex · 2026-11-11
- Freezing the VLM costs π models 23-56 points but barely hurts GR00T — rahulshivchand · 2026-11-11
- Stronger zero-shot ≠ faster adaptation: GR00T N1.5 beats N1.7 in fine-tuning — rahulshivchand · 2026-11-11
- New paper benchmarks pi vs GR00T: how many demos does a VLA actually need to learn a task? — rahulshivchand · 2026-11-11
- Meituan LongCat's dNaViT shows discrete codes work as universal visual representations — meituan-longcat · 2026-11-11
- ExRL Boosts Sample-Efficient RL Adaptation of Long-Chunk Robot Policies via Execution Length Control — Xu Luo · 2026-11-11