Backprop-free pretraining hits 4,000x speedup in 10 days, now 1.6 seconds per run
SonglinYang4 · x · 2026-10-30
Zeroth-order optimization is having a breakthrough month: in the 10 days since the Noise speedrun launched, transformer pretraining time fell from 1.76 hours to 1.6 seconds (4,000x). Latest record: 3.5 val loss on FineWeb-30M in 22.7s on one 8xH100 node (down from 123s). Key wins: single-root credit assignment (pairing was pure estimator noise), Gluon Hopper TMA kernels, and fused chain tails keeping 64-move tiles in shared memory to avoid 1GB HBM round trips. Scaling remains the open question—some tricks won't scale, and the team plans dataset scaling tests starting at 30M tokens.
More from Infra
- Claude Duet says it runs on Xeon, Dot on EPYC — Ben Bajarin asks if the backends are fungible — BenBajarin · 2026-10-30
- DCOCO paper: contextual correction makes parallel speculative decoding up to 6.6x faster — burny_tech · 2026-10-30
- Big Tech FCF fell from $60.7B to $7.4B as capex doubled; self-cannibalization is rational — JFPuget · 2026-10-30
- A 512GB Mac Studio is cheap: 90s SGI seats cost $100k in 1990s dollars — dreamwieber · 2026-10-30
- Amazon Principal Scientist Ido Rosen: Unit Economics First, Open Weights Only Months Behind — arena · 2026-10-30
- Intel signals 2027 capex above estimates, no positive FCF expected next year — BenBajarin · 2026-10-30