Self-Play Pretraining with Zero Data: Models Learn from Self-Generated Data Alone

Researchers from Stanford's Noah Goodman group, Tel Aviv University and others (Michael Y. Li et al., with Cowsik, Dolev, De Luca, Cohen, Goodman, Levine) present a proof-of-concept study, "Self-Play Pretraining with Zero Data": two models both start from random initialization — a generator proposes programs for a universal Turing machine that output byte sequences, and a learner trains only on those outputs. The current conclusion is that with self-generated data alone, the learner improves predictably on diverse natural-data prediction tasks as compute grows, and in-context learning (ICL) emerges.

Confirmed

Training uses zero real-world data: the learner sees only self-generated byte sequences, with no natural datasets or human-curated corpora.
Core finding: as self-play compute increases, the learner's ability to predict real text, images, speech, audio and melodies improves, with validation loss falling predictably — scaling is predictable.
ICL emerges after training purely on self-generated data.

Why it matters

The experiment challenges the default assumption that pretraining requires large-scale natural corpora, suggesting that compute plus a self-play mechanism can yield statistical structure transferable to natural data.
As a proof of concept, it offers a new direction for training when data is scarce, and raises foundational questions about the relationship between data, compute and learning.

The work remains a proof of concept; scaling paths and upper bounds await further validation.

2026-09-25 ~ 2026-09-27 · 14 related posts

Primary sources

New paper shows models can learn general patterns from zero real-world data — mark_k · 2026-09-25 23:46 (UTC+08:00)

[source] New paper shows models can learn general patterns from zero real-world data — mark_k · 2026-09-25
Zero-data pretraining: two LMs bootstrap all training data via self-play from random init — ChengleiSi · 2026-09-26
Self-Play Pretraining with Zero Data: Models Learn From Self-Generated Compute Alone — burny_tech · 2026-09-27
Self-Play Pretraining with Zero Data: models learn without any human data — burny_tech · 2026-09-27
Stanford's Self-Play Pretraining: LMs learn from zero real data — StanfordAILab · 2026-09-27

9 near-duplicate retellings: inductionheads · ctjlewis · inductionheads · AlexTensor · burny_tech · burny_tech · burny_tech · burny_tech · CatAstro_Piyush