Pretraining gains come mostly from data: experiments show 12x vs 3.7x compute multipliers
eliebakouch · x · 2026-09-09
Dwarkesh Patel and a collaborator pretrained combinations of year-representative open model recipes and data corpora from 2019 to 2025 at various small scales to quantify where pretraining progress comes from.
- Data improvements delivered 3.24x as many compute multipliers as model improvements (12.0x vs 3.7x).
- The gains stack independently: a better dataset helps every architecture roughly equally, and vice versa.
- Full results and implications for the future of AI progress are published in a writeup.
The finding pushes back on the narrative that architecture innovation is the main engine of pretraining progress.
Related event: Dwarkesh Experiments: Data Drives Most Pretraining Progress(3 posts)→
More from Models
- Only 2 of 25 full ChatGPT rewrites flagged 100% AI by detectors — paulnovosad · 2026-10-26
- Qwen 4 Reportedly Spotted in the Arena, Hints at Alibaba's Next-Gen Model Testing — AIFlow_ML · 2026-10-26
- Nathan Lambert's first open models lecture: distillation, risks and the ecosystem explained — natolambert · 2026-10-26
- Nanbeige 4.3-33B flops real-world test: zero compilable output vs Qwen — julianharris · 2026-10-26
- Claude's safety classifier blocks SS7 defense research, pushing a security dev to an abliterated GLM — l4rz · 2026-10-26
- Black Forest Labs open-sources 5.6B FLUX 3 Image Klein, +91 Elo with 38% fewer params — dl_weekly · 2026-10-26