无反向传播预训练追平 backprop,8×H100 上 2.5 分钟达 val 3.704
tensorqt · x · 2026-10-22
- Emanuele Rodăola reports forward-only pretraining now matches backprop on the Noise benchmark: val 3.704 in 2.5 minutes on 8×H100, vs 3.712 for backprop with the same model, data and optimizer; the previous record (Boost→Cut) was 3.945 in 7.6 min. No autograd anywhere.
- Key techniques in the update: data-free start (block 1's heads initialized as lag-1 RoPE heads), a tail average of weights, skipping attention credit for 4 blocks late in the run, overlapped collectives, and a compiled forward pass.
- Follow-up claims training without a backward pass is now faster than backprop overall.
「Infra」频道最新
- 洪堡大学 PACE:4-bit 量化线性注意力状态,免微调降显存 — huberlin · 2026-10-22
- Hugging Face 上架 Kernels 官方教程：一行代码给模型换上预置 GPU 算子 — ariG23498 · 2026-10-22
- NVIDIA 联手沙特 HUMAIN 建最高 84MW 算力,约 2.7 万颗 GPU — HUMAIN · 2026-10-22
- 黄仁勋：能源是 AI 最大瓶颈，现在是投资一切能源形式的时候 — kimmonismus · 2026-10-22
- NVIDIA 发布 Vera Rubin 全栈 agent 硬件，Nemotron 后训练宣称超越前沿模型 — mervenoyann · 2026-10-22
- 梗图：ASML 独扛现代数字基建，微软 Teams 被调侃成威胁行为者 — VoidAsuka · 2026-10-22