Zero-data pretraining: self-play models learn from scratch with predictable scaling

AlexTensor · x · 2026-09-27

A proof-of-concept called Self-Play Pretraining with Zero Data: a generator and a learner both start from random init; the generator proposes programs for a universal Turing machine and the learner trains on their outputs, with no real data used. Zero-shot val loss on images, text, audio and melodies decreases predictably with self-play compute, and the learner develops in-context learning. Pedro Domingos notes it fits his "deep networks are kernel machines" view.

Related event: Self-Play Pretraining with Zero Data: Models Learn General Patterns from Self-Generated Data Alone(9 posts)→

Original post →

More from Research

• MIT economist challenges Aaru's human-simulation benchmarks: opaque method, no baseline, data leakage risks — soumitrashukla9 · 2026-09-27
• DeMiAn: dense language annotations boost robot policy learning, cut compute 62% — rajammanabrolu · 2026-09-27
• UC Berkeley opens tenure-track faculty position in AI for Biology — anshulkundaje · 2026-09-27
• Single neuron sufficient to bypass safety alignment in LLMs, paper finds — amplifiedamp · 2026-09-27
• C5R's SciUniverse benchmark exposes AI failures at the lab bench — VraserX · 2026-09-27
• Interpretability researcher: sandbagging signals from probes would block model deployment — thebasepoint · 2026-09-27

Research channel →