Study questions synthetic pre-pretraining: never beats natural data under compute parity

A preprint from Cohere, UCL and MIT researchers finds that synthetic-data pre-pretraining never outperforms pure natural language training in compute-matched experiments, with apparent gains traced to embedding initialization bias rather than the synthetic data itself.

2026-11-03 ~ 2026-11-04 · 2 related posts

• Compute-matched study: synthetic pre-pretraining never beats natural language — LauraRuis · 2026-11-03
• Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9% — LauraRuis · 2026-11-04