Transformer pretraining without backprop drops from 1.76 hours to 1.6 seconds in 10 days
mayfer · x · 2026-10-23
A viral optimization thread claims the past month may be the biggest in zeroth-order optimization history: since the Noise speedrun released, transformer pretraining time fell from 1.76 hours to 1.6 seconds (4,000x), and the Dust method was up to 10,000x more compute-efficient than prior SOTA. This shows transformers can be pretrained without backprop — and far more efficiently than expected. Scaling is the big open question: some tricks won't scale, so the team will grow the dataset (from 10M to 30M tokens and beyond) whenever it overfits to discard them.
Related event: Forward-only pretraining surges: 1.76 hours to 1.6 seconds in 10 days(2 posts)→
More from Infra
- Only 8% of cloud vendors support OIDC — dev calls to kill long-lived API keys — cramforce · 2026-10-23
- Swift-1.5-Qwen3.8-27B ported to hipfire MQ4: 2.2x faster end-to-end than GGUF on a 7900 XT — Queasy_Asparagus69 · 2026-10-23
- Intel CTO: chip output can at best 2X while AI compute grows 3X per year — SammaVaco · 2026-10-23
- Engineer reverse-engineers projector TV's 10s wake lag to a cold-booting Linux board — banteg · 2026-10-23
- Ollama CEO says enterprise AI is having its PC moment on This Week in AI — ollama · 2026-10-23
- Modal open-sources Spindle, a Tinker-compatible training server that charges GPU time instead of tokens — josh_wills · 2026-10-23