Zero-data self-play pretraining: generator-learner pairs learn from scratch and develop in-context learning
CatAstro_Piyush · x · 2026-09-27
A proof-of-concept called Self-Play Pretraining with Zero Data: two models start from random initialization — a generator proposes programs for a universal Turing machine, and a learner trains on their outputs, with no real data used at all.
Key findings:
- Zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute
- The learner develops in-context learning capabilities
Co-led by michaelyli with AdityaCowsik and KfirDolev, with Noah Goodman and Yoav Levine among co-authors. Another researcher applied a similar generator-language approach to vision models, producing line and circle drawings with varied placement, rotation, and intensity after only 250 self-play rounds.
More from Research
- Mobile AI Factory: agentic pipeline builds on-device SLMs that beat 5-shot frontier models — kalyan_kpl · 2026-11-11
- Amazon paper: single agents beat multi-agent setups on enterprise SOPs at 1.4–4.7x lower cost — rohanpaul_ai · 2026-11-11
- Amazon paper: multi-agent setups burn 1.4-4.7x tokens for small gains; single agent is the default — rohanpaul_ai · 2026-11-11
- MIT's Buehler builds "recursive meta-intelligence": hundreds of AI agents inhabit self-created worlds to study material failure — ProfBuehlerMIT · 2026-11-11
- Explosion launches Ellf directory: 52 NLP templates plus multi-agent annotation workflow — _inesmontani · 2026-11-11
- STABLE Optimizer Accepted at NeurIPS 2026: Taming Continual Learning Forgetting at the Optimizer Level — v_lomonaco · 2026-11-11