Q Labs pretrains Transformers without backprop using randomized forward evals, scaling to 243M params
jiqizhixin · x · 2026-11-07
Q Labs' Dust technical report demonstrates GPT-style Transformer pretraining on FineWeb without a backward pass: parameter updates are estimated from loss changes across randomized forward evaluations, and evolutionary search is moved from weight space to activations.
- Scale: verified up to 243M parameters, with direct pretraining budgets of up to 20M tokens.
- Mechanism: each token gets an independent perturbation at a linear layer's output, building a virtual population without copying the whole model; perturbations earn credit when loss improves, including weighted effects on later tokens; reward-weighted noise estimates the error signal, combined with layer inputs to update weights.
- Search: single draws reach 1,448 perturbation samples per step.
- Tradeoff: larger populations improve gradient alignment and training quality but cost substantially more compute; main comparisons use separately tuned momentum SGD.
More from Research
- AI-generated peer reviews at ICLR/NeurIPS punish papers for stating their own limitations — serrjoa · 2026-11-07
- PSI says its internal system, on a non-public Anthropic model, proved the 2008 Aaronson-Ambainis conjecture — DeryaTR_ · 2026-11-07
- Perplexity reading-group slides connect modern model design to fast weights, SSMs — PMinervini · 2026-11-07
- Poincar3: 3D Self-Supervised Learning via Multi-View Self-Distillation — rsasaki0109 · 2026-11-07
- Incoming ML masters student: hundreds of papers read, still no elegant research ideas — sooham · 2026-11-07
- JITEngine synthesizes custom LLM inference engines for Apple Silicon, claims up to 2.7x speedup — AccBalanced · 2026-11-07