"Deep lifts" paper: ReLU networks adaptively generalize even with the dumbest random optimizer
yuxiangw_cs · x · 2026-11-04
A new arXiv paper by Sunil Madhow and Yu-Xiang Wang asks how prevalent "good weights" are in large neural networks — the theory behind why even the dumbest optimizer generalizes adaptively.
- Setup: sample uniformly at random from parameters of a width-W, depth-D ReLU network that interpolate the data, when the data is actually explained by a smaller w-by-d network.
- Result: despite the oblivious selection rule, the sampled θ̂ generalizes adaptively at rate Õ(wdWD/n), not governed by the VC dimension of the full W×D class.
- Mechanism: a w×d witness implies an exponentially large volume of well-generalizing W×D interpolating parameters; the authors construct it via a proof technique called "deep lifting" and tie volume to stability.
- Implications: approximation-theoretic results convert into generalization guarantees for nonparametric classes, and uniform-random interpolators can even survive label corruption.
In short: good weights are everywhere in large networks — even random interpolators hit them.
More from Research
- TAMU professor releases 400+ page AI-assisted tutorial deck on generative models — ShuiwangJi · 2026-11-04
- DeepMind mafia seeding a new wave of AI for Science startups in the UK — ersatzben · 2026-11-04
- MIT CSAIL fixes small models by hand-editing weights — hype or old idea rebranded? — ziv_ravid · 2026-11-04
- Robots need failure data: learned sims handle rendering, MuJoCo keeps contact honest — chris_j_paxton · 2026-11-04
- Robotics researcher: training neural sims needs ~10x more failure than success data — chris_j_paxton · 2026-11-04
- Hugging Face adds native benchmark leaderboards and eval results to the Hub — NielsRogge · 2026-11-04