Same nets, 3 seeds: Adam hits 0.033 test loss vs SGD's 0.442 on regression
cneuralnetwork · x · 2026-10-23
The author ran controlled PyTorch experiments comparing Adam vs SGD on regression and classification, keeping initial weights identical, sweeping learning rates, and repeating across 3 random seeds.
Key findings
- Adam converged much faster on both tasks; on regression it reached 0.033 test loss vs SGD's 0.442 — over an order of magnitude better
- Learning-rate sweeps show SGD plateauing at high loss across the tested range while Adam drops steadily
Why it happens
- SGD scales every gradient with one learning rate: θₜ₊₁ = θₜ − ηgₜ
- Adam keeps moving averages of gradients and squared gradients, updating with bias-corrected estimates: θₜ₊₁ = θₜ − ηm̂ₜ/(√v̂ₜ + ε)
- Per-parameter adaptive step sizes let Adam handle directions with very different gradient magnitudes
The author notes Adam isn't always better, but the optimizer alone dramatically changes training dynamics.
More from Research
- How a 0.1 microsecond rounding error made the Patriot missile system miss a Scud after 100 hours — Franc0Fernand0 · 2026-10-23
- Papers with Code auto-links methods and benchmarks across 121k papers for just $5 — NielsRogge · 2026-10-23
- Language as a Sensor: CoRL 2026 Paper Fuses Speech With Vision for Robot Mapping — andreea7b · 2026-10-23
- Viral claim: Stanford finds dopamine-like reward neurons emerging inside LLMs — ashajadeja · 2026-10-23
- Qwen open-sources BlenderWorld, a 223-task benchmark testing if agents can build editable 3D scenes — petewoodbridge · 2026-10-23
- AI is revealing the space of mathematical ideas as negatively curved, researcher argues — prof_g · 2026-10-23