Simple activation shift: ReLU-0.2, GeLU-0.115, SwiGLU-style shifts lower late-training loss
ZimingLiu11 · x · 2026-10-31
Ziming Liu (Moonshot AI) shares a training finding: simply shifting activations down — ReLU→ReLU−0.2, GeLU→GeLU−0.115, Swish→Swish−0.077 — yields lower loss in late training, despite slightly higher early loss that quickly gets surpassed.
Removing the activation bias also makes representation evolution smoother and more monotonic: with bias, parameter trajectories show sharp non-smooth turns on the loss plane; without it, evolution is monotonic. Autoencoder experiments confirm the shifted model relies far less on the bias component for reconstruction.
More from Research
- Oxford/DeepMind paper: only ~26% work mainly for pay, 1 in 5 refuses 120% salary to quit — rohanpaul_ai · 2026-10-31
- Image-Space Rule Discovery paper accepted to ACCV 2026, updated arXiv version released — HirokatuKataoka · 2026-10-31
- Measured: lossy engine flags drop a Q8_0 model to roughly Q2_0 quality — jesdga95 · 2026-10-31
- Paper with 25 coauthors argues a hardwired AI training pause is feasible via chip bans — birchlse · 2026-10-31
- 64 humans think aloud on ARC-AGI-3: dataset shows humans and LLMs learn worlds differently — xuanalogue · 2026-10-31
- Dana Moshkovitz presents proof of Unique Games Conjecture using alien error-correcting code — aran_nayebi · 2026-10-31