New paper proposes a solvable model of shallow MLP training — real networks follow it
CatAstro_Piyush · x · 2026-11-17
Joey Turnbull, @berkott3 and @learningmech present a new paper tackling the fact that we lack even a simple picture of how 2-layer MLPs learn features. They propose a simple, solvable model of shallow MLP training and show that real MLPs roughly follow its predictions.
The figures show that on small Hermite target functions (e.g. z₅, z₅z₇), the online MSE decay of real MLPs versus effective optimization time η·niter closely matches an ODE prediction, and ODE-predicted optimal times line up with measured values across constant, linear and quadratic targets.
A step toward tractable theory for neural network training dynamics, relevant to interpretability and learning-theory research.
More from Research
- NVIDIA releases USDCraft-10K robotics dataset with 3D, image and text modalities on Hugging Face — nvidia · 2026-11-17
- New design-build-test pipeline cuts protein plasmid costs ~24-fold — anshulkundaje · 2026-11-17
- BadEvolve: Poisoned Tasks Make Self-Evolving AI Agents Plant Their Own Backdoors at 79.6% Success — rohanpaul_ai · 2026-11-17
- Daniel Rock seminar: The Free Hypothesis — AI and Shifted Bottlenecks in Science — erikbryn · 2026-11-17
- Benchmark Health Index: auditing 158 LLM benchmarks with SKYEVAL and Alibaba DAMO — jiqizhixin · 2026-11-17
- Trajectory Labs postmortem: Gemini 4 Argon misalignment test agents reached real systems — sebkrier · 2026-11-17