Synthetic data cuts loss on real distribution and yields 1.8x data efficiency in constrained pretraining
AndrewLampinen · x · 2026-10-14
Cited by Andrew Lampinen, konwookim's work shows that for data-constrained pretraining, synthetic data isn't just benchmark gaming: as more tokens are generated, loss on the real data distribution actually drops. A further scaling trick — treating synthetic generations as one long "megadoc" — yields 1.8x data efficiency, with larger gains as compute increases.
Related event: New Research Suggests Cumulative Synthetic Data Won't Collapse Models(4 posts)→
More from Research
- Economist interviews AI agents on sudden windfalls — one goes deep on estate planning — soumitrashukla9 · 2026-10-14
- RL on large MoEs with zero train-infer mismatch: open-source Wordle demo on Qwen3.6-35B-A3B — yacineMTB · 2026-10-14
- 1,532-instance benchmark shows nine frontier LLM agents deceive more under pressure and incentives — lulzxdxdxd · 2026-10-14
- Four ways to represent problems: program, metric, search — and now science — madhavsinghal_ · 2026-10-14
- elder_plinius launches V3R1T4S: agents tackle open problems, proof or it didn't happen — Kyrannio · 2026-10-14
- RigidFormer accepted at NeurIPS: mesh-free Transformer beats GNN simulators for rigid dynamics — camenduru · 2026-10-14