Economists keep getting overparameterization wrong: SGD's implicit regularization is the point
Afinetheorem · x · 2026-09-07
Economist Jesús Fernández-Villegas pushes back on the common economist complaint that deep neural networks 'require regularization,' noting economics regularizes all the time—control variables, Bayesian priors, splines, kernels, sieves—just implicitly and without cross-validation. Large neural networks make regularization implicit: SGD picks the minimum-norm solution among those fitting the training data. The thread also highlights Mahdi Kahou's joint work 'The Blessings of Overparameterization,' showing heavy overparameterization reduces dependence on initialization.
More from Research
- SC Asia 2027 opens call for papers on supercomputing and AI infra, due Oct 7, 2026 — thoefler · 2026-09-07
- Ex-Google Brain researcher: key algorithmic wins were found under 1e20 FLOPs, then scaled to 1e25 — _arohan_ · 2026-09-07
- GlossoGen paper: LLM agents evolve emergent languages humans can't understand — abenitezburraco · 2026-09-07
- New paper resolves three open problems in online fair division with impossibility results — chaumian · 2026-09-07
- Pedro Domingos: Recurrence Over Input Sequences Is Bad, Recurrence in Inference Depth Is Good — pmddomingos · 2026-09-07
- New Research Tackles LoRA's Random Down-Projection Bottleneck for Faster Fine-Tuning — burkov · 2026-09-07