Transformer pretraining without backprop gets 4,000x faster in 10 days, beats Adam on validation loss
burny_tech · x · 2026-10-23
- Zeroth-order optimization just had a landmark month: 10 days after the Noise speedrun release, transformer pretraining time dropped from 1.76 hours to 1.6 seconds — a 4,000x speedup, with Dust up to 10,000x more compute-efficient than the prior state of the art.
- The team claims transformers can now be pretrained without backprop and even beat backprop+Adam on validation loss: "search fking works."
- Scaling is the biggest open question — some tricks behind the 1.6-second result won't scale; the plan is to grow the dataset each time it overfits, starting from 30M tokens.
- Ironic twist: the capability breakthrough was inspired by a safety blog.
More from Research
- AnyGoal research asks whether world models memorize trajectories instead of learning reusable rules — randall_balestr · 2026-10-23
- Stratified Sparse Attention unifies retrieval and sampling to close the sparse-to-dense gap at extreme sparsity — Skylight1 · 2026-10-23
- New paper with 25 coauthors examines feasibility of a hardwired pause on frontier AI training — wfithian · 2026-10-23
- Tadpole: TUM uses autoencoders as foundation models for 3D PDEs with online learning — jiqizhixin · 2026-10-23
- FDA crosses a regulatory watershed with personalized neoantigen cancer vaccine, per Gottlieb — alex_peys · 2026-10-23
- VIPerturb-seq Update: 10x Genomics to Commercialize Single-Cell CRISPR Screening Tech — anshulkundaje · 2026-10-23