Signature Emerges After Just 2-4 SGD Updates as Weights Shift ~8%
maier_ak · x · 2026-10-28
The author shares first-hand observations on weight permutation symmetry in neural networks:
- A model signature appears after only 2 to 4 SGD updates, by which point weights have moved roughly 8%.
- Git Re-Basin aligns better than the author's own 2019 method.
- Limitations: only one hidden layer, three similar datasets, and no attractor demonstrated.
A link to the full write-up is included.
More from Research
- DFR Framework Tackles Reward Fine-Tuning in Diffusion Models Without Losing Diversity — mmbronstein · 2026-10-28
- alphaXiv ships Mobile 2.0 with AI paper reader and one-click visual blogs — datawithsuman · 2026-10-28
- Researchers screened 175 Spanish books ahead of time and found multiple AI-generated positives — TuhinChakr · 2026-10-28
- Xiaohongshu and Fudan Propose DR-Rubric: Agent-Written Rubrics as RL Rewards — jiqizhixin · 2026-10-28
- Ben Goertzel: AI Peer Review Already Beats Humans, and Human Bias Is Harder to Fix — bengoertzel · 2026-10-28
- Developer Joins DeepMind's Antigravity Science Program, Claims Gemini Argon Is Frontier in Biomedicine — DeryaTR_ · 2026-10-28