Noise: a speedrun for pretraining transformers without backprop
madhavsinghal_ · x · 2026-10-13
- The industriaalist team launched Noise, a speedrun tool for pretraining without backprop, arguing zeroth-order optimization replacing backprop is inevitable.
- Last week they presented Dust, the first method to approach backprop with SGD for pretraining transformers, and believe another 1,000x improvement is possible.
- Getting there requires co-designing credit assignment, optimizers, architectures, and eventually hardware—Noise exists to iterate faster on this direction.
More from Infra
- Foxconn slides reveal Omniverse digital twins and GR00T humanoids building AI servers — firstadopter · 2026-10-13
- LiquidAI ships MLX versions of d1 models optimized for Apple Silicon local inference — JosephJacks_ · 2026-10-13
- Apple's new Private Cloud Compute release ships vserver images, hinting at virtualized macOS servers — jedisct1 · 2026-10-13
- HF's Zach Mueller: don't do this — most users can just rent or run sandboxes locally on CPUs — TheZachMueller · 2026-10-13
- Celesto CI undercuts GitHub Actions 9x: 1000 jobs for $6.67 vs $60 — aniketmaurya · 2026-10-13
- Keurig coffee maker flooded Wi-Fi with 1TB in 10 days — owner says unit bug — gnukeith · 2026-10-13