Stanford's Self-Play Pretraining: LMs learn from zero real data

StanfordAILab · x · 2026-09-27

Stanford researchers introduce Self-Play Pretraining with Zero Data: two models start from random initialization — a generator proposes programs for a universal Turing machine, a learner trains on the outputs, with no real data ever used.
Despite never seeing real data, the learner shows predictable scaling: zero-shot val loss on images, text, audio, and melodies decreases with self-play compute, and it develops in-context learning abilities.
Theoretical motivation: Transformers can be trained to approximate Solomonoff induction, the ideal next-token predictor, so learning to predict natural sequences without data access is possible.
Framed as a proof-of-concept, co-led by AdityaCowsik, michaelyli, and KfirDolev, with Noah Goodman and Yoav Levine among co-authors.

More from Models