Backprop-Free Pretraining Gets 4000x Faster in 10 Days, Rivaling Adam

Zeroth-order optimization hits a milestone: within 10 days of the Noise speedrun launch, backprop-free Transformer pretraining dropped from 1.76 hours to 1.6 seconds—about 4000x faster—with validation loss beating Adam and forward-only training matching backprop on 8×H100.

2026-10-22 ~ 2026-10-23 · 4 related posts

• Forward-only pretraining matches backprop: val 3.704 in 2.5 min on 8×H100, no autograd — tensorqt · 2026-10-22
• Transformer pretraining without backprop drops from 1.76 hours to 1.6 seconds in 10 days — mayfer · 2026-10-23
• Q Labs pretrains transformer without backprop: 4,000x speedup to 1.6s in ten days — rickasaurus · 2026-10-23

1 near-duplicate retellings: burny_tech