Researcher argues synthetic data can't enable recursive self-improvement: compression loses information
gerardsans · x · 2026-10-21
On X, Gerard Sans pushed back on the idea that synthetic data is a runway for recursive self-improvement (RSI): training is compression, compression implies data loss, and re-sampling cannot create new information or expand the data's support.
His alternatives:
- Prefer high-quality curated pre-training data that expands support
- Otherwise use models with non-overlapping training corpora
- If you must use the same model family, mix synthetic with real data and monitor data loss and overfitting
More from AGI Musings
- AI-native law firms' own lawyers shun internal tools for Claude Code, hinting BigLaw will just acquire them — jkubicki · 2026-10-21
- Alpha School staffer stunned by his 4-year-old's math gains in six weeks — nateliason · 2026-10-21
- Why AI Companies Are Betting Personality Beats Commoditized Intelligence — The AI Daily Brief · 2026-10-21
- 'We don't eat pigs because we're smarter': AI doomer analogy sparks pushback — theteknosaur · 2026-10-21
- Rudd claims China leads AI adoption, but US revenue data says otherwise — ShakeelHashim · 2026-10-21
- Researcher questions Nature for publishing AI consciousness debate, points readers to SEP instead — nanjiang_cs · 2026-10-21