Self-Play Pretraining with Zero Data
Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, Yoav Levine
cs.AI, cs.CL
2026-09-25
A generator writes programs for a universal Turing machine and a learner predicts their outputs, both from random init. With zero natural data, zero-shot loss on text, images, and audio falls as a power law in compute.
Progress in language modeling has come from pretraining on ever more data, but that data is still curated on the model's behalf: corpus filtering, mixture design, hand-built synthetic generators. The ceiling on data supply is human knowledge and effort. This paper (Stanford, Tel Aviv University, and collaborators) tests the extreme version of the alternative: give the model zero natural data and let it generate its own training set. The question is whether anything learned that way transfers to real text, images, and audio. It does, and the transfer follows a power law.
Two identical decoder-only transformers, both randomly initialized. A generator writes programs in a Brainfuck-style language (eight instructions plus ten single-character macros) for a universal Turing machine; executing a program on a random input tape emits a byte sequence, and a learner trains on those sequences with plain next-byte cross-entropy. Everything is byte-level with a 256-symbol vocabulary, and neither model ever sees natural data.
The interesting part is how the generator is trained. Its reward is learning progress: the gradient a program's output induces in the learner, scored by how well it aligns with the learner's parameter movement over the past half of training, measured under an AdamW preconditioner. Programs the learner has mastered produce near-zero gradients; unlearnable noise produces gradients that point nowhere useful. Both get nothing. Only content at the frontier of what the learner can currently absorb keeps earning reward, so the curriculum upgrades itself as the learner improves. A pure difficulty reward fails here: random bytes injected into a predictable sequence are arbitrarily hard and completely useless.
Engineering scaffolding: each round mixes fresh, mutated, and replayed programs for exploration and replay-based retention; the policy gradient uses GRPO-style batch normalization with clipped importance ratios; reward-weighted supervised fine-tuning (expert iteration) guards against forgetting. The execution semantics make every string a valid program, so the generator can never crash the machine.
Zero-shot validation loss (no gradient step ever touches the eval sets) follows L(C) = E + A·C^-b across text, images, audio, speech, melody, math, DNA, and code. Effective compute counts ensemble size × params × tokens, spanning roughly 10^15 to 10^19, with models below 25M parameters and a per-run budget of 34.36B tokens.
Exponents, self-play versus published natural-data values:
| Dataset | Self-play b | Literature b |
| DCLM text | 0.123 | 0.048–0.099 |
| CIFAR-10 (HWC) | 0.066 | 0.065–0.10 |
| 16-bit audio | 0.141 | 0.12–0.14 |
| Metamath | 0.129 | 0.17 |
| C source | 0.116 | 0.17 |
| DNA (8-symbol) | 0.435 | 0.01–0.06 |
Broadly comparable, often a bit higher. DNA is the lone anomaly, an order of magnitude above the literature range, and the paper flags it without explaining the mechanism.
Two controls show the adaptive curriculum is doing the work. Sampling programs from a fixed Solomonoff-style prior over the same program space scales far more slowly, so access to a universal program space alone is not enough. PCFG pretraining, a hand-designed source of hierarchical structure, still wins on text and code but loses on images, audio, and melody, and its models are strong only on associative recall in the ICL evaluations.
The generator is visibly finding structure. Across 1.64×10^8 uniformly sampled programs, only arithmetic sequences appear (1,526 hits); Fibonacci, geometric, quadratic, and cubic sequences get zero hits, implying a first appearance beyond round 53,000. Self-play finds all five families by round 512. The generator also keeps improving: corpora built from generator checkpoints at increasing endpoints, each used to train a fresh 1M-parameter learner, show rising epiplexity and falling out-of-distribution validation loss with later endpoints. The learner picks up in-context learning for free: reverse string, stack, and associative recall approach 100% accuracy under greedy decoding with enough examples; on byte summation, it gets the low 4 bits right after about 4 examples and the high bits by 8.
Practically, a self-play checkpoint works as pre-pretraining. Warm-starting a 24.4M model reaches converged loss on ESC-50 with 320M tokens versus 496M from random initialization, and on CIFAR-10 with 421M versus 588M; the warm start hits every loss level first, with the gap narrowing near convergence. The compute spent on self-play itself is not charged to this comparison.
Synthetic data is usually treated as an amplifier for natural corpora. This paper sits at the other endpoint of the design space: data generated purely by compute. It splits the Chinchilla data term into two resources, contingent information and universal predictive structure, and argues that only the second is being grown here, which explains why synthetic data far from the target distribution can still reduce loss on it. Even if universal pretraining never replaces real data, it quantifies how much transferable structure can be bootstrapped from nothing, with scaling curves clean enough to reason over. At 25M parameters and 4K context this is a controlled experiment rather than a recipe; as a controlled experiment, with both models starting from random weights, any transfer must come from the self-play process itself.
The authors' own list: all experiments sit below 25M parameters at 4K context, and scaling up is unverified and may require a more expressive programming language; contingent facts about our world cannot be generated from nothing, so this cannot replace natural data; hyperparameters and model selection used validation loss on DCLM and DNA, a limited leak of natural data into the pipeline; the pre-pretraining comparison does not charge the self-play compute.
Two things stand out on a close read. The DNA exponent (0.435 against a literature range of 0.01–0.06) is marked as exceptional but never explained, and DNA happens to be one of the two datasets in the model-selection signal. And the reading of matched exponents, that learning universal structure accounts for a large component of natural-data scaling, rests on an ansatz-driven argument in which the observed data exponent approximates the smaller of two decay rates; the authors hedge this explicitly. Whether the discovered mathematical sequences causally drive transfer is untested, with circuit analysis and curriculum ablations named as future work.