Hessian eigenvalue plots show removing activation bias flattens the loss landscape
ZimingLiu11 · x · 2026-10-31
Part 6/N of the same research thread: experiments plotting Hessian eigenvalues at training steps 0/5000/10000 show that removing the activation bias cuts top eigenvalues from 20-50 down to 2, flattening the loss landscape throughout training.
This explains why optimization no longer suffers from sharp "valley" directions that cause oscillations and loss spikes. An auto-encoder experiment additionally shows the representation's monotonic evolution is restored once the bias is removed.
More from Research
- Overfitting explained with two students and two exams — cneuralnetwork · 2026-10-31
- Betelgeuse runs 68GB Qwen MoE bit-exact at 22 tok/s on RTX 3060 + 16GB RAM — zyxciss · 2026-10-31
- EdgeRock-4B open-sourced with fixed 72MiB KV cache, 99.1% less memory than base — testingcatalog · 2026-10-31
- Claude's Muon optimizer theorem checks out numerically to 4 decimal places — vxnuaj · 2026-10-31
- Dev asks for a mathematician: Claude's Muon optimizer theory is too deep to verify — vxnuaj · 2026-10-31
- Zero-mean 'bug' in ReLU/SiLU/GeLU found to cause loss spikes; fixing it boosts LLMs and ViTs — ZimingLiu11 · 2026-10-31