Zeroth-order optimization papers need retesting on Dust; pretraining is the holy grail
burny_tech · x · 2026-10-22
apazcli notes many zeroth-order optimization papers show similar results via different perturbation-direction tricks, and these should be adapted to Dust and re-tested—likely without moving the needle, but worth setting research agents on. The holy grail is solving it for large networks in a pretraining context; we still don't understand what happens to loss landscapes that makes SPSA-like algorithms fail.
More from Research
- DeepMind AI Agent Autonomously Discovers and Lean-Verifies Rules for 11 Unsolved Turing Machines — rohanpaul_ai · 2026-10-22
- New Paper Derives Exact Stability Criterion for Outer Optimizers in Communication-Efficient Distributed Training — apsarathchandar · 2026-10-22
- Karpathy open-sources AutoResearch: AI agents that run the full research loop autonomously — mdancho84 · 2026-10-22
- MF² benchmark at EMNLP 2026 tests VLMs' movie comprehension with 850+ fact/fib pairs — andre_t_martins · 2026-10-22
- LLM literature priors boost Bayesian optimization: reaction yields jump from 6% to 39% — bravo_abad · 2026-10-22
- An 11-step AI workflow to extract data from papers for literature review — Faheem_uh · 2026-10-22