New Research Suggests Cumulative Synthetic Data Won't Collapse Models

Andrew Lampinen highlighted multiple studies countering fears of model collapse from synthetic data: collapse doesn't occur when data accumulates, synthetic data can cut real-data needs by 1.8x, and self-improvement methods like STaR continue to strengthen models.

2026-10-14 ~ 2026-10-14 · 4 related posts

• Study: model collapse doesn't happen when synthetic data accumulates, even without new real data — AndrewLampinen · 2026-10-14
• Synthetic data cuts loss on real distribution and yields 1.8x data efficiency in constrained pretraining — AndrewLampinen · 2026-10-14
• Google researcher: having models write new training data via ICL improves finetuning generalization — AndrewLampinen · 2026-10-14
• Revisiting STaR: models self-improve by training only on rationales that reach correct answers — AndrewLampinen · 2026-10-14