Dust paper: zeroth-order optimization pretrains Transformers, 1000x efficiency gain

The Dust paper is the first to pretrain Transformers with zeroth-order optimization, matching or beating backprop with up to 1000x efficiency. Developers report driving validation loss below 4 within days and have shared a leaderboard.

2026-10-16 ~ 2026-10-17 · 2 related posts

• Pretraining without backprop: Dust hits sub-4 loss in days via associative credit memory — madhavsinghal_ · 2026-10-16
• Dust paper: first zeroth-order method to pretrain transformers, 1000x over SOTA ES — burny_tech · 2026-10-17