Q Labs pretrains transformer without backprop: 4,000x speedup to 1.6s in ten days
rickasaurus · x · 2026-10-23
Q Labs' Noise speedrun community cut no-backprop transformer pretraining time to a fixed validation-loss target (4.2) from 1.76 hours to 1.6 seconds in ten days — over 4,000x faster — and the latest run beats backprop + Adam on validation loss.
- Dust algorithm was already up to 10,000x more compute-efficient than prior state-of-the-art zeroth-order methods
- A chart shows stacked optimizations (compiled probes, full Hadamard cycles, fused kernels, sharding, restarts, fine-tuning) on a single 8xH100 node driving the curve down to seconds
- Biggest open question is scaling: some tricks won't scale, so the team will grow the dataset on overfit, starting at 30M tokens
More from Infra
- HarnessRouter open-sources a unified API to run Codex, Claude Code and more as agent backends — tom_doerr · 2026-10-23
- Local AI security checklist: pin Ollama ≥0.35.0, vLLM ≥0.31.0, purge LiteLLM 1.107.0 malware — AIFlow_ML · 2026-10-23
- Open-source Bonsai 2 9B gets 471% faster on Macs after one-week community optimization challenge — gajesh · 2026-10-23
- July 28 MCP release makes remote servers stateless, biggest revision since launch — bibryam · 2026-10-23
- MCP keynote: AAIF hits 305 members, Skills over MCP and Tasks/Events roadmap revealed — bibryam · 2026-10-23
- Fizgig 8.9.0 adds multi-GPU LoRA training across PCs and rented pods — shootthesound · 2026-10-23