Removing activation bias is a free trick that improves ViT, DiT and GPT training
ZimingLiu11 · x · 2026-10-31
A near-zero-cost training trick: removing the activation bias (shift on activations) yields consistent gains across ViT image classification, DiT image generation, and GPT language pretraining.
- Loss residual plots show the shifted model consistently outperforming baseline, with suppressed hidden-state overshoot in ViT-B.
- From a loss landscape perspective, removing the bias substantially lowers Hessian eigenvalues throughout training, yielding a flatter landscape and avoiding sharp "valley" directions that cause oscillations and spikes.
- The change is essentially free, per the author.
Paper link in the original thread.
More from Infra
- Betelgeuse runs 68GB Qwen MoE bit-exact at 22 tok/s on RTX 3060 + 16GB RAM — zyxciss · 2026-10-31
- ~25 Internet-Facing Palo Alto Firewalls Rooted One Week After php2shell Patch — cyb3rops · 2026-10-31
- French lawmakers approve €0.055 per gigabyte tax on internet traffic — helloplanets · 2026-10-31
- Joke: why steal catalytic converters when unmanned Waymos carry $150k+ in GPUs — vincent_koc · 2026-10-31
- Stuck at 120-160 tps on local LLM: Reddit debates AVX-512 vs AVX2 for CPU pick — catbluesgames · 2026-10-31
- Rumors say Google is killing AI Overview, citing ad click damage and compute strain — Dan_Jeffries1 · 2026-10-31