Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9%
LauraRuis · x · 2026-11-04
A preprint by researchers from Cohere, UCL AI Centre and MIT (Stern, Rocktäschel, Locatelli, Ruis) re-examines synthetic pre-pretraining (PPT) across three synthetic curricula (k-Dyck, NCA, declarative reference corpus) and 50B-token training runs. In compute-matched experiments PPT never beats pure natural-language training. A single change — initializing token embeddings from N(0,0.02) instead of N(0,1) — improved baseline validation perplexity by 9.0%, fully wiping out the 5.4% gain previously attributed to PPT on synthetic cellular automata. An alternative scheme improves both methods but narrows the gap, and giving the natural-language budget extra compute persists past 1B tokens while PPT gains disappear. The authors conclude synthetic PPT's benefits as a distinct stage are limited and such data should be reconsidered.
More from Research
- Cambridge's van der Schaar Lab opens PhD applications for 2027 entry — MihaelaVDS · 2026-11-04
- Tencent's Hyra 2.0 Generates Protein Sequences 57x Faster Than ProteinMPNN — YvesMulkers · 2026-11-04
- Researcher Urges Top AI Conferences to Desk-Reject AI-Generated Papers Amid Fake Data Scandals — Vegetable-Formal-753 · 2026-11-04
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04
- kalomaze shares loss curve of larger-backbone run on DeepSeek V3.1 base, eval loss down to ~0.82 — kalomaze · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04