Percolation Dynamics in Optimization: Variance Cascades in SGD, Adam and AdamW
Sai Niranjan Ramachandran · hf · 2026-09-04
SGD dynamics are modeled as a percolation process where architectural symmetries cause subnetworks to merge in discrete blocks, producing variance spikes resembling phase transitions; similar trapping behavior under heavy-tailed noise extends to Adam and AdamW.
More from Research
- AI BioDesign accelerator launches with UW Medicine and Fred Hutch to let AI design biology — AllThingsApx · 2026-09-04
- Sarvam AI releases Vāgartha: 217,959 Sanskrit verses with detailed explanations on Hugging Face — selfawareatom · 2026-09-04
- ORNL's AI system arranges molecules atom-by-atom, working 25+ hours unattended — Scobleizer · 2026-09-04
- zackmdavis pitches embryo selection as the no-GPUs path to smarter humans, citing Gwern — zackmdavis · 2026-09-04
- OpenAI researcher: GPT-6's CoT controllability keeps rising over RL training — gleech · 2026-09-04
- Compile by Training: Distilling Natural-Language Specs into Local Neural Functions — UWaterloo · 2026-09-04