Data improvements deliver 12x compute multipliers vs 3.7x for model changes
josh_wills · x · 2026-11-11
Dwarkesh Patel highlighted a study (overshadowed by the Navier-Stokes news) in which he and @whoisjerbear pretrained combinations of year-representative open model recipes and data corpora from 2019–2025 at various small scales.
Key findings:
- Data improvements contributed 12.0x in compute multipliers vs 3.7x for model/architecture improvements — a 3.24x gap
- The gains stack independently: better datasets help every architecture about equally, and vice versa
- The authors argue this is strong evidence that data, not architecture, is the true major source of algorithmic progress in pretraining, with implications for how AI progress continues
More from Research
- NVIDIA's ADEPT Accepted at CoRL 2026: Sim-Pretrained Dexterity Deploys Zero-Shot on Real Robots — ankurhandos · 2026-11-11
- SynthonMap: a zoomable map of billion-to-trillion-scale chemical spaces — leland_mcinnes · 2026-11-11
- Sara Seager and son challenge 'water chauvinism': sulfuric acid as alien life's solvent — _onionesque · 2026-11-11
- David Bau's Cornell Tech talk on interpretability, indirection and introspection now on YouTube — yoavartzi · 2026-11-11
- Cisco releases Signals 2.0 to auto-diagnose 45 types of AI agent failures, hitting 91.43% F1 — ptkbhv · 2026-11-11
- Review of 6 RSI benchmarks finds flawed tasks in 5, with 31-63% error rates — teortaxesTex · 2026-11-11