Forward-only pretraining matches backprop: 3.704 val in 2.5 min on 8×H100, no autograd
tensorqt · x · 2026-10-20
Emanuele Rodolà reports that forward-only pretraining now matches backprop on the Noise benchmark: the Relay method hits val 3.704 in 2.5 minutes on 8×H100, vs 3.712 for backprop with the same model, data, and optimizer — and beats the previous record (Boost→Cut, 3.945 in 7.6 min). No autograd anywhere. Commenters note his habit of casually dropping results like this mid-conversation.
More from Research
- UCSB's REAL AI lab hiring postdocs to build autonomous labs spanning AI, chemistry and robotics — ninamiolane · 2026-10-20
- Schaeffer wraps pretraining work: generative eval scaling laws and data repetition — RylanSchaeffer · 2026-10-20
- He pitched pretraining-inference compute exchange rate research 9 months before o1 — RylanSchaeffer · 2026-10-20
- Gold-likelihood predicts GSM8K pass rates better than compute scaling laws — RylanSchaeffer · 2026-10-20
- Accumulating data avoids model collapse, and an untold AI-agents backstory — RylanSchaeffer · 2026-10-20
- Researcher: universal multimodal jailbreaks seem near-impossible across VLMs — RylanSchaeffer · 2026-10-20