LLM training speedup 19% by switching to built-in GELU
rasbt · x · 2026-08-20
A case study by Giles Thomas shows that replacing a hand-rolled GELU activation function with PyTorch's built-in version increased LLM training speed from 21,000 tokens/second to 25,000 tokens/second.
This demonstrates that prioritizing optimized functions provided by deep learning frameworks can yield significant performance gains outside of educational contexts.
More from Infra
- Mac can now run a 27B model locally that codes, reasons, and sees — TheMoonMidas · 2026-08-20
- RTX PRO 6000 Blackwell Max-Q Review: Ideal for Multi-GPU Towers — TheZachMueller · 2026-08-20
- Dory: An open-source alternative to OrbStack and Docker Desktop for Mac — tom_doerr · 2026-08-20
- Custom LLM Inference Engine: 10-20% Faster than vLLM — AlpinDale · 2026-08-20
- Running a 35B MoE at ~60 tok/s on a 12GB RTX 4070 Ti: full config — Seraphym87 · 2026-08-20
- Musk on SpaceX: The 4% Margin for Multi-Planetary Life — r0ck3t23 · 2026-08-20