Linear mode connectivity appears on test loss but not train loss, notes researcher Arohan
_arohan_ · x · 2026-09-28
Researcher arohan calls linear mode connectivity one of the most underrated features of training neural nets, recalling his 4-year-old finding that it appears on test loss but not on train loss — something he found mind-blowing at the time.
Updates from @stanislavfort's colab using DistributedShampoo (near-zero train loss): train/test loss of 0.0002/0.314 vs baseline 0.350/0.333, and train/test accuracy of 0.999/0.982 — showing the phenomenon persists even in the low-train-loss regime.
Related event: Linear Mode Connectivity Sparks Debate Over Its Origins(2 posts)→
More from Research
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- Kind ICLR Reviewer Raises Score Despite Unresolved Concerns, Citing the Paper's Value — MarkSchmidtUBC · 2026-11-16
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16