Zeroth-order pretraining 4000x faster in 10 days, but 99.9% of directions never explored, warns practitioner
burny_tech · x · 2026-10-25
- The quoted post calls it possibly the biggest month in zeroth-order optimization history: since the Noise speedrun release, backprop-free transformer pretraining dropped from 1.76 hours to 1.6 seconds (4000x), and Dust was up to 10,000x more compute-efficient than prior SOTA. Scaling remains the big open question; the team will grow the dataset whenever it overfits, starting from 30M tokens.
- Francois Chauba1 pushes back: ZO has no gradient flow and needs architectural change, and after two years he only made it work for fine-tuning, never pretraining. The core issue is dimensionality: training an 8B model for 100k steps with 64 perturbations checks only 6 million halfspaces—under 0.1% of directions, leaving 99.9% unexplored and gradient noise too high late in training. Early progress looks good only because all directions give finite differences initially.
More from Infra
- 305M-Parameter Sparse MoE Runs Computer-Use Agent Fully Offline on a MacBook Air — lmoroney · 2026-10-25
- SGLang reruns DeepInfer on matching hardware, matches or beats its prefill performance — hsu_byron · 2026-10-25
- Anthropic in talks for $45B compute deal with Nscale for 460MW of Vera Rubin capacity — Beth_Kindig · 2026-10-25
- DIY GPU cluster war stories: PCIe retimers can literally catch fire — giffmana · 2026-10-25
- GPT-6.1 Sol beats Opus 5.5 on cost-performance balance, argues dev who caught Opus in logic traps — haider1 · 2026-10-25
- Abliterated DeepSeek V4.1 Flash GGUF hits Hugging Face trending — orcarouter · 2026-10-25