1T tokens of experiments: synthetic data least useful in pretraining, COLM paper finds
edwardbeeching · x · 2026-10-14
A COLM Outstanding Award paper by Joel Niklaus and Hugging Face researchers (with Colin Raffel, Leandro von Werra, Thomas Wolf) systematically studies how to synthesize high-quality pretraining data:
- Over 1T tokens generated, 90 experiments, 1.9B documents analyzed across rephrasing strategy, generator model, and source data.
- Structured output formats (tables, math problems, FAQs, tutorials) consistently beat curated web baselines and prior synthetic methods.
- Scaling the generator model beyond 1B parameters yields no additional benefit.
- Choice and mixing of source data substantially affects performance.
- They release FinePhrase, an open 486B-token dataset of rephrased web text that outperforms all synthetic baselines while cutting generation costs up to 30x.
- The headline hot take after all that: synthetic data is the least useful in pretraining.
Paper, blog, pipeline code, and dataset are all open-sourced.
More from Research
- Embedding arithmetic demo: cartoon − animation − comic = diagram — mayfer · 2026-10-23
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23