Dust paper: first zeroth-order method to pretrain transformers, 1000x over SOTA ES
burny_tech · x · 2026-10-17
The Dust paper introduces the first zeroth-order method to pretrain transformers, approaching and sometimes exceeding backprop with heavy compute. Key claims: 1,000–10,000x more compute-efficient than EGGROLL, the state-of-the-art ES method; chosen because pretraining transformers is arguably the hardest task for zeroth-order optimization; and high-dimensional search improves with overparameterization at fixed population size. A researcher digesting it notes many implementation details are unexplained and is reverse-engineering the math in a blog post.
More from Research
- MSR: Test-Time Method Externalizes Diversity to Corpus-Derived Sampling for Generative Models — Kostas Triaridis · 2026-10-17
- Scott Aaronson shows Rosser's theorem via Turing machines, no arithmetic encoding needed — jessi_cata · 2026-10-17
- Interpretability researchers debate: is adding a neuron quadratic in parameters? — BlancheMinerva · 2026-10-17
- Gödel's flip side: addition and multiplication alone make math Turing complete — jessi_cata · 2026-10-17
- Replicating an interpretability paper with Opus 5.5: 20 of 22 results confirmed on Qwen3-32B — Sauers_ · 2026-10-17
- Knock out neurons, score traits: a reproducible protocol for persona attribution — Sauers_ · 2026-10-17