Zero-data pretraining: two LMs bootstrap all training data via self-play from random init

ChengleiSi · x · 2026-09-26

Researchers introduce Self-Play Pretraining with Zero Data: two models start from random initialization — a generator proposes programs for a universal Turing machine, and a learner trains on their outputs, never touching real data.

Key findings:

Zero-shot validation loss on natural images, text, audio, and melodies decreases predictably with self-play compute
The learner spontaneously develops in-context learning capabilities

Co-led with Aditya Cowsik and Kfir Dolev, with Noah Goodman and Yoav Levine among co-authors, the work is a proof of concept suggesting LMs can in principle bootstrap their own pretraining data.

Original post →

Zero-data pretraining: two LMs bootstrap all training data via self-play from random init

More from Research