Zero-Data Pretraining: Random-Init Models Generate All Training Data via Self-Play
burny_tech · x · 2026-09-27
Researchers from Stanford AI Lab and collaborators (with AdityaCowsik, KfirDolev, gbrunodl, Noah Goodman, Yoav Levine) introduce Self-Play Pretraining with Zero Data.
- Two models start from random initialization: a generator proposes programs for a universal Turing machine, and a learner trains on their outputs — no real data used at all.
- As self-play compute scales, zero-shot validation loss on images, text, audio, and melodies decreases predictably, and the learner develops in-context learning capabilities.
- The authors frame it as a proof-of-concept exploring whether pretraining can work with zero real data.
More from Research
- Explosion launches Ellf directory: 52 NLP templates plus multi-agent annotation workflow — _inesmontani · 2026-11-11
- STABLE Optimizer Accepted at NeurIPS 2026: Taming Continual Learning Forgetting at the Optimizer Level — v_lomonaco · 2026-11-11
- From Bahdanau to FlashAttention: a timeline of attention's 8-year evolution — cneuralnetwork · 2026-11-11
- Three years building CPUs and spiking nets: one engineer's path to AI accelerators — k7agar · 2026-11-11
- When AI becomes an oracle, what happens to mathematics? — stevenstrogatz · 2026-11-11
- 200 Years of Linear Algebra in One Chart: From Leibniz's Determinants to Peano's Vector Spaces — burny_tech · 2026-11-11