Self-Play Pretraining with Zero Data: Models Learn From Self-Generated Compute Alone
burny_tech · x · 2026-09-27
A paper from Tel Aviv University, Stanford and others, "Self-Play Pretraining with Zero Data", proposes pretraining without human-curated data: two models train in tandem from random initialization. A generator proposes programs interpreted by a universal Turing machine to emit byte sequences, while a learner predicts those sequences with standard cross-entropy. The generator is updated via RL to target sequences at the frontier of the learner's ability—neither solved nor random—forming an adaptive curriculum, effectively searching the space of all computable structure (inspired by Solomonoff induction).
Remarkably, despite never seeing natural data, the learner improves zero-shot on unseen text, images, audio and code, and even develops in-context learning. Zero-shot loss scales predictably with self-play compute, suggesting general predictive structure can be discovered from self-generated computation alone, making training data limited by compute rather than human knowledge.
More from Research
- Scandinavian Journal of Statistics editors reflect on publishing in the age of generative AI — michaelchchoi · 2026-11-11
- Proving LLM backdoor-safety without revealing weights: ZKML over formal methods — DimitrisPapail · 2026-11-11
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- cloneofsimo releases a video explaining determinantal point processes — cloneofsimo · 2026-11-11