Shifting activation bias slightly (ReLU−0.2, GELU−0.115) lowers late-stage training loss
ZimingLiu11 · x · 2026-10-31
Ziming Liu shares a subtle but effective training tweak: apply a small negative bias to common activations (ReLU→ReLU−0.2, GeLU→GeLU−0.115, Swish→Swish−0.077) to get lower loss late in training.
The accompanying figure shows non-zero activation bias creates pathological loss landscapes: hidden-state means visibly overshoot around step 100 with non-smooth trajectories, slowing late-stage convergence. The shifted model learns slower initially but soon catches up and surpasses, reaching lower final loss across all three activation functions.
More from Research
- Image-Space Rule Discovery paper accepted to ACCV 2026, updated arXiv version released — HirokatuKataoka · 2026-10-31
- Measured: lossy engine flags drop a Q8_0 model to roughly Q2_0 quality — jesdga95 · 2026-10-31
- Paper with 25 coauthors argues a hardwired AI training pause is feasible via chip bans — birchlse · 2026-10-31
- 64 humans think aloud on ARC-AGI-3: dataset shows humans and LLMs learn worlds differently — xuanalogue · 2026-10-31
- Dana Moshkovitz presents proof of Unique Games Conjecture using alien error-correcting code — aran_nayebi · 2026-10-31
- MetaCircle Launches: Building AI That Discovers the Hidden Laws of AI Itself — ZimingLiu11 · 2026-10-31