Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs
inductionheads · x · 2026-10-18
Inspired by Nick Cammarata's interpretability essay "Neural Networks Are Already Clean," a developer applied its ideas to speed up backprop-free pretraining, achieving a 2.3x speedup with the new Noise method (still in progress).
Context: the industriaalist team recently released Dust, the first method to approach backprop with SGD for pretraining transformers, and believes another 1,000x improvement is possible via co-designing credit assignment, optimizers, architectures, and hardware. The chart shows Boost→Cut reaching Dust's 72-minute validation loss (val 3.991) in just 7.6 minutes on 8 GPUs.
More from Research
- Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
- semiconductor.diy: open-source guides to make real chips in a garden shed — jwt0625 · 2026-10-18
- MIT builds multimodal AI that sees 3D crystal structures to speed materials discovery — MIT News AI · 2026-10-18
- Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
- q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18
- Smarter Every Day's Destin to defend PhD dissertation on impact fusion — CatAstro_Piyush · 2026-10-18