Networks Trained on the Same Task Find Related Weights Within Four Updates
maier_ak · x · 2026-10-28
A new paper in Frontiers in Cognition trained 30 small networks on MNIST, Fashion-MNIST and KMNIST, removed neuron-permutation and ReLU-scaling symmetries, then compared weights pair by pair.
Key finding — "clustering, not convergence": same-dataset networks end up 3x more similar than cross-dataset ones (0.079 vs 0.238). Same images and init but a different rule (digit vs parity) also separates them: the objective shapes the weights, not just the data.
The signature appears after only 2–4 SGD updates, while weights have moved only 8%. Git Re-Basin aligns better than the authors' 2019 method. Limits: one hidden layer, three similar datasets, no attractor shown.
More from Research
- CPR-Dataset: retrievers fail at cross-domain discovery, training gives +70% on BRIGHT — coallaoh · 2026-10-28
- Hugging Face robot duck walks in simulated snow with 290,000 physics-simulated flakes — lukas_m_ziegler · 2026-10-28
- Mercor, Stanford and NBER launch TaskNet, an open-source task catalog updating O*NET for the AI era — soumitrashukla9 · 2026-10-28
- François Fleuret Proposes Tagging Every Token 'Heard/Thought/Said' to Solve Alignment — francoisfleuret · 2026-10-28
- Reddit builds benchmark release-date timeline showing evals go stale after 3-4 months — Express_Quail_1493 · 2026-10-28
- VibeVideoBench: multimodal agents coding videos often fail on time management — Muxi Diao · 2026-10-28