Prediction: zeroth-order optimization methods may eventually replace backprop
j_foerst · x · 2026-10-07
A thread concludes that despite sampling-based methods (dust) still being far costlier than backprop, the bitter lesson will hit optimization research too: zeroth-order methods — the simplest form of optimization — may eventually replace backprop entirely, enabling training of non-differentiable architectures and finding better loss-landscape directions than first-order gradients.
More from Research
- MEMOIR-VLM hits 91.3% balanced accuracy on dementia classification with missing brain-scan inputs — PTenigma · 2026-10-07
- Claude paper's prompter was an Anthropic employee, authors were just digesters, says user — analisereal · 2026-10-07
- Claude math result row: prompter was an Anthropic employee, authors say they were the "digesters" — analisereal · 2026-10-07
- ChipForge NPU Challenge Goes Live: Crowdsource AI Accelerator Designs Headed to Real Silicon — bittingthembits · 2026-10-07
- SIGGRAPH Asia 2026 Lineup: Disney's Markus Gross, Huawei's Ran Huang, Oxford's Andrea Vedaldi — AlexTensor · 2026-10-07
- CoreWeave RL Rollouts Hot-Loads Policy Weights Into Live Deployments, ~15x Faster Than Redeploys — _ScottCondron · 2026-10-07