Researchers debate: can heuristics replace gradient updates in overparameterized nets?
jonasgeiping · x · 2026-10-23
NYU researcher Jonas Geiping and giffmana debate whether neural nets require gradient-based training. giffmana offers an optimistic counter: with the right inductive biases and a well-organized parameter space (like the Transformer), things could be far less pessimistic. Geiping responds that overparameterized nets can indeed be updated with tailored heuristics — but the flip side applies: if the problem were that well-conditioned, why not use gradients, the only tool that updates a high-dimensional model in all directions simultaneously?
More from Research
- Is formally verifying an AI agent sandbox a Millennium Prize-level problem? — teortaxesTex · 2026-10-23
- General Reasoning's RLCR trains agent organizations with collective rewards — hierarchies emerge — daniel_mac8 · 2026-10-23
- JUICE: one flow-matching Transformer unifies image and text generation, hitting 0.88 GenEval — pratyusha_PS · 2026-10-23
- DeepMind's Verena Rieser to keynote EMNLP 2026 on a science of constitutional alignment — verena_rieser · 2026-10-23
- Anthropic uses Claude to write 180M lines of proofs, verifying 123 open-source libraries — The Decoder · 2026-10-23
- SciLib: A Lean-Based Framework Brings Machine-Checkable Formal Verification to Scientific Theories — CatAstro_Piyush · 2026-10-23