Linear mode connectivity appears on test loss but not train loss, notes researcher Arohan
_arohan_ · x · 2026-09-28
Researcher arohan calls linear mode connectivity one of the most underrated features of training neural nets, recalling his 4-year-old finding that it appears on test loss but not on train loss — something he found mind-blowing at the time.
Updates from @stanislavfort's colab using DistributedShampoo (near-zero train loss): train/test loss of 0.0002/0.314 vs baseline 0.350/0.333, and train/test accuracy of 0.999/0.982 — showing the phenomenon persists even in the low-train-loss regime.
More from Research
- Mathematician prototypes Lean-based tool to teach high school Euclidean geometry with proofs — AlexKontorovich · 2026-09-28
- Columbia's SPEAR framework reframes AI alignment as an ongoing interactive process after deployment — windx0303 · 2026-09-28
- SAGE Uses Topological Guidance to Fix Long-Horizon LLM Reasoning Biases — Xinyue Zeng · 2026-09-28
- LightMIS: 0.13M-Param Medical Segmentation Net Cuts 99% Params, Matches Accuracy — Andrei Arhire · 2026-09-28
- Eval tips for automated optimizers: use AUC over accuracy, sample 10x and check consensus to cut hallucinations — a_karvonen · 2026-09-28
- Three Simple Inference Tricks Significantly Improve Activation Oracle Results — a_karvonen · 2026-09-28