Self-Play Pretraining with Zero Data: models learn without any human data
burny_tech · x · 2026-09-27
A new paper, "Self-Play Pretraining with Zero Data" (Cowsik, Dolev, Li, De Luca, Cohen, Goodman, Levine et al.), pretrains models entirely via self-play on generated byte sequences with no human data.
- Models show predictable zero-shot scaling and in-context learning on natural datasets despite never seeing human data
- Suggests foundational capabilities can emerge purely from discovering computable structure via compute
The authors argue this reshapes views on synthetic data, curriculum learning, and what fundamentally drives pretraining representations. Rated a "certified banger" (86/100) by arXivBangers.
More from Research
- Mobile AI Factory: agentic pipeline builds on-device SLMs that beat 5-shot frontier models — kalyan_kpl · 2026-11-11
- Amazon paper: single agents beat multi-agent setups on enterprise SOPs at 1.4–4.7x lower cost — rohanpaul_ai · 2026-11-11
- Amazon paper: multi-agent setups burn 1.4-4.7x tokens for small gains; single agent is the default — rohanpaul_ai · 2026-11-11
- MIT's Buehler builds "recursive meta-intelligence": hundreds of AI agents inhabit self-created worlds to study material failure — ProfBuehlerMIT · 2026-11-11
- Explosion launches Ellf directory: 52 NLP templates plus multi-agent annotation workflow — _inesmontani · 2026-11-11
- STABLE Optimizer Accepted at NeurIPS 2026: Taming Continual Learning Forgetting at the Optimizer Level — v_lomonaco · 2026-11-11