Pretraining without backprop: Dust hits sub-4 loss in days via associative credit memory
madhavsinghal_ · x · 2026-10-16
jaydenteoh reports pushing Dust — the first zeroth-order method approaching SGD for pretraining transformers — below 4 validation loss within days.
- Associative-memory zeroth-order optimization: use past (input, gradient) pairs as associative memory to provide a control-variate guess, updated via least-squares mesa-optimization, cutting update variance.
- Orthogonal noise draws for probing.
- Key insight: Dust perturbs activation space, not weight space — much lower dimensional.
- The original team argues replacing backprop is inevitable, with another 1000x improvement possible via co-designing credit assignment, optimizers, architectures and hardware.
More from Research
- Captive owls recognize familiar caregivers' voices and their relationship quality — Sauers_ · 2026-10-17
- PhD defence slides visualize a PhD as 30,635 Slack messages across projects — aryaman2020 · 2026-10-17
- Fully funded PhD position in NLP for indigenous languages in Mexico City — m2saxon · 2026-10-17
- Stanford NLP PhD aryaman joins Anthropic's Interpretability team — aryaman2020 · 2026-10-17
- Sooool: latency-aware agent plays Touhou in real time, repo and guide open-sourced — wzenus · 2026-10-17
- Training neural nets with interpretability as a hard constraint — why hasn't anyone tried? — inductionheads · 2026-10-17