Compute-matched study: synthetic pre-pretraining never beats natural language
LauraRuis · x · 2026-11-03
A new preprint led by Alex runs compute-matched experiments on pre-pretraining (PPT) with synthetic distributions like k-dyck and NCA, and finds PPT never outperforms pure natural-language training. Even when language data is scarce (100M tokens), repeating epochs over a natural-language corpus beats every synthetic curriculum. The results directly challenge the popular practice of allocating pre-training compute to synthetic data. Details in the linked thread.
More from Research
- CMU: Model Merging and 'Grafting' Let Post-Trained LLMs Learn Off-Policy Without Forgetting — AdtRaghunathan · 2026-11-03
- IBM Research hiring 2027 summer interns for AI-driven protein binding mutation prediction — pinyuchenTW · 2026-11-03
- EPFL's SOMA benchmark shows multimodal models far behind humans on body-motion understanding — amathislab · 2026-11-03
- Berkeley's GEPA-Curator filters noisy failure signals to boost agent prompt optimization — Berkeley · 2026-11-03
- HKUST's AuditFL adds a blockchain clearing layer for auditable federated learning — HKUST · 2026-11-03
- Skill-centric training beats task-centric for small web agents across 4 benchmarks — anmarasovic · 2026-11-03