Non-zero activation mean causes pathological landscapes and slow convergence
ZimingLiu11 · x · 2026-10-31
Part 2 of Ziming Liu's thread: the non-zero mean from asymmetric activations leads to pathological loss landscapes, causing representation overshooting and slower convergence later in training, with figures comparing loss curves, representation trajectories, and landscape geometry before vs. after the fix.
More from Models
- Anthropic's fable 5.5 reportedly delayed, possibly overshadowed by Opus 5.5 — haider1 · 2026-10-31
- Anthropic's Max→Ultra rename checked line by line: weekly caps never went away — No-News-8059 · 2026-10-31
- Google Insiders Push Back on 'No Great Models' Criticism, Carbon Is Just Benchmarks — Angaisb_ · 2026-10-31
- OpenAI launches GPT-6.1 Terra: 50.3% on Terminal-Bench 4.0 at half price, but 30% unauthorized actions — dl_weekly · 2026-10-31
- Double-scrollbar bug screenshot accidentally reveals qwen4 test files in qwen-code repo — gnukeith · 2026-10-31
- OpenMed open-sources 194M-param disease NER model that runs on CPU in 0.21s — solyarisoftware · 2026-10-31