Near-identical neural net weights still diverge on ~3% of inputs after training
bravo_abad · x · 2026-11-08
Ly and Gong's study explains why deep learning is so sensitive to tiny changes: two networks starting from almost identical weights under the same deterministic training procedure still disagree on roughly 3% of inputs, and shrinking the initialization gap barely reduces this.
They trained VGG convolutional nets with SGD on handwritten digits (half the labels deliberately corrupted); prediction disagreement stayed near 2.9% as initialization differences varied across ten orders of magnitude.
The proposed explanation comes from dynamical systems: starting points leading to different outcomes are interwoven at extremely fine scales, with fractal-like phase portraits and a power-law fit supporting this geometry in a minimal network.
More from Research
- Removing negative-advantage samples drops robot zip-tie success from 86% to 60% — micoolcho · 2026-11-08
- DECIDE benchmark tests voice-agent decisions on 4,867 questions from 1,081 real calls, scoring 20 models — Aiden_Tech_Ai · 2026-11-08
- KV Cache Quantization Tested Across 35 Quants: 8-bit Is Essentially Free for Qwen3.8 27B — Hefty_Wolverine_553 · 2026-11-08
- YC amplifies demo of models handling semi-serious X-band phased-array design — ycombinator · 2026-11-08
- Analysis of 250k ICLR reviews finds 65% AI-written, scores collapsing to middle — TuhinChakr · 2026-11-08
- Israeli researchers report method letting AI teach itself without human data — windless12 · 2026-11-08