Forward-only pretraining matches backprop: val 3.704 in 2.5 min on 8×H100, no autograd
tensorqt · x · 2026-10-22
- Emanuele Rodăola reports forward-only pretraining now matches backprop on the Noise benchmark: val 3.704 in 2.5 minutes on 8×H100, vs 3.712 for backprop with the same model, data and optimizer; the previous record (Boost→Cut) was 3.945 in 7.6 min. No autograd anywhere.
- Key techniques in the update: data-free start (block 1's heads initialized as lag-1 RoPE heads), a tail average of weights, skipping attention credit for 4 blocks late in the run, overlapped collectives, and a compiled forward pass.
- Follow-up claims training without a backward pass is now faster than backprop overall.
More from Infra
- PACE: Post-Hoc 4-Bit Quantization of Linear-Attention Recurrent States Without Fine-Tuning — huberlin · 2026-10-22
- Hugging Face ships official Kernels tutorial: drop prebuilt Hub kernels into your PyTorch model — ariG23498 · 2026-10-22
- HUMAIN and NVIDIA plan up to 84MW of AI compute in Saudi Arabia, ~27,000 GPUs — HUMAIN · 2026-10-22
- Jensen Huang: Energy is AI's biggest bottleneck, time to invest in all forms of energy production — kimmonismus · 2026-10-22
- NVIDIA unveils Vera Rubin full-stack agent hardware, claims post-trained Nemotron beats frontier models — mervenoyann · 2026-10-22
- Meme: all of modern digital infrastructure rests on one ASML machine — VoidAsuka · 2026-10-22