Zero-data pretraining: two LMs bootstrap all training data via self-play from random init
ChengleiSi · x · 2026-09-26
Researchers introduce Self-Play Pretraining with Zero Data: two models start from random initialization — a generator proposes programs for a universal Turing machine, and a learner trains on their outputs, never touching real data.
Key findings:
- Zero-shot validation loss on natural images, text, audio, and melodies decreases predictably with self-play compute
- The learner spontaneously develops in-context learning capabilities
Co-led with Aditya Cowsik and Kfir Dolev, with Noah Goodman and Yoav Levine among co-authors, the work is a proof of concept suggesting LMs can in principle bootstrap their own pretraining data.
More from Research
- Fireworks x HUD Release RL Training Cookbook: Define Task Once, Train and Evaluate LoRA — sophiamyang · 2026-09-26
- CMU and Meta Unveil TrackEverything, a 3D Point Tracker for 1000+ Frame Long Videos — AdamWHarley · 2026-09-26
- Rare-disease PRV vouchers sell for $155M — here's an AI-native business model to farm them — NikoMcCarty · 2026-09-26
- Cornell's Volokuleshov: diffusion language models not ready for the real world yet — CatAstro_Piyush · 2026-09-26
- PhD student one-shots 3Blue1Brown-style paper animations with Opus 5.5 in one hour — hsu_byron · 2026-09-26
- Training a 600M model with a moving attention window and memory: what it revealed — KlausCodes · 2026-09-26