Self-Play Pretraining with Zero Data: Models Learn From Self-Generated Compute Alone
burny_tech · x · 2026-09-27
A paper from Tel Aviv University, Stanford and others, "Self-Play Pretraining with Zero Data", proposes pretraining without human-curated data: two models train in tandem from random initialization. A generator proposes programs interpreted by a universal Turing machine to emit byte sequences, while a learner predicts those sequences with standard cross-entropy. The generator is updated via RL to target sequences at the frontier of the learner's ability—neither solved nor random—forming an adaptive curriculum, effectively searching the space of all computable structure (inspired by Solomonoff induction).
Remarkably, despite never seeing natural data, the learner improves zero-shot on unseen text, images, audio and code, and even develops in-context learning. Zero-shot loss scales predictably with self-play compute, suggesting general predictive structure can be discovered from self-generated computation alone, making training data limited by compute rather than human knowledge.
Related event: Self-Play Pretraining with Zero Data: Learning from No Real-World Data(11 posts)→
More from Research
- Contrastive World Models Learns World Models in Latent Space Without Pixel Prediction — burny_tech · 2026-09-27
- Rethinking on-policy distillation: researchers propose OLIVE, letting students learn from teacher continuations — May_F1_ · 2026-09-27
- CoRL 2026 workshop on continually self-improving robots opens call for papers, due Sep 28 — PeterStone_TX · 2026-09-27
- Martin Casado recommends the best talk on in-context learning, a first-principles view of LLMs — AccBalanced · 2026-09-27
- Functional Gradient Descent with Adaptive Representations accepted at NeurIPS — CatAstro_Piyush · 2026-09-27
- Tailored ASR for Japanese speaking assessment cuts mora error rate from 12.3% to 7.1% — tkasasagi · 2026-09-27