Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus, Michal Štefánik, Aline Villavicencio, Nikolaos Aletras
cs.CL, cs.AI, cs.LG
2026-09-30
Retested across 500M-7B models and up to 100B tokens, synthetic pre-pretraining still saves 21B tokens at 3B, but its gains come from long-range retrieval, not a grammatical prior.
Pre-pretraining (PPT) is a recent warm-up trick: a randomly initialized model trains for a few hundred steps on synthetic non-natural-language sequences, then carries those weights into regular pre-training (PT). Prior work reported token-efficiency gains and attributed them to a "grammatical prior," a structural bias picked up from bracket languages that supposedly transfers to natural-language grammar.
That conclusion rested on three conditions far from real pre-training. Models never exceeded 1B parameters. PT stopped before 2B tokens, so the gains could be a transient initialization effect. And the corpora were almost pure web text, while modern recipes mix in code and math, domains already full of nested structure that might make PPT redundant. A team from the University of Sheffield, Hugging Face, MBZUAI, and NII scaled all three variables up and retested.
The study is a four-way grid: 5 PPT tasks × 4 PT data mixtures × 4 scales (500M/1B/3B/7B) × two budgets (a standard 21B tokens, extended to 100B). The standard budget alone exposes 12.8 to 25.6 times more tokens than prior work; the extended budget is roughly 50×.
Control separates the synthetic data itself from the mere 500 extra steps of optimization: if warm-up steps were what mattered, Control should gain too. Results average over checkpoints 5K-10K rather than a single snapshot, with a gain counting as stable only when its mean exceeds its SD. The appendix justifies this: single-checkpoint BLiMP estimates flip sign in half the pairs, while downstream averages never do.
| Setting | Metric | Result |
| 12 scale-mixture pairs (k-Shuffle Dyck) | downstream avg change | 9 pairs gain at least +0.6; mean +1.6 among them |
| 3B Marin | tokens saved | ≥21B: PPT hits 62.3 at 63B tokens; PT-Only reaches 62.0 only at 84B |
| 100B tokens (3B) | downstream avg | +1.6 on C4, +1.2 on Marin, level with 21B |
| 7B Marin (75.5B tokens) | downstream avg | +0.6, positive at every budget point |
| BLiMP grammar test | stable gains | 1/12 pairs (downstream avg: 10/12) |
| Verbatim retrieval | NLL drop | 12/12 pairs, stable in 7 |
| Set task (3B, 4 mixtures) | downstream avg | -7.2, worse on all 4 mixtures |
| Math share 1.3%→17% | PPT gain | +2.0 vs +1.9, essentially unchanged |
| Web text removed entirely | PPT gain | collapses from +1.9 to +0.2 |
Three findings stack up.
The gains are real and they scale. They hold steady across sizes (500M +0.8, 1B +1.4, 3B +1.3), Control sits within ±0.4 of PT-Only at every scale, and all 14 extended budget points stay positive, so longer training does not dilute the effect.
The grammatical-prior explanation fails. Only 1 of 12 pairs shows a stable BLiMP gain, and the exact setting of prior work (C4, 500M) actually loses 0.6. On Marin the BLiMP delta swings from +0.3 at 500M to +1.6 at 1B and back to -0.3 at 3B. The reliable signals are verbatim retrieval (improved in all 12 pairs) and the three benchmarks gaining most downstream: LAMBADA, ReCoRD, and HellaSwag, all of which require information from the preceding passage.
Task ablations pin down the mechanism. k-Shuffle Dyck (+1.3), NCA (+1.2), and MP-Struct Core (+0.9) gain comparably; Set alone loses 7.2 points. The three effective tasks all force the model to locate a specific earlier position (a closing bracket finds its opener; NCA reproduces the matching cell of the previous state), while Set only asks whether a token appeared before, with no position to find. Data-side ablations agree: a 13× rise in math share leaves the gain untouched, and removing web text collapses it. Web share cannot explain OLMo3's failure either: the 17% Math variant matches its 76.9% web share yet gains +2.0, against +0.0 for OLMo3 at 3B.
For pre-training practitioners the accounting is favorable: roughly 5% of training time at the 3B/21B setting (about 1% against a 100B budget) buys at least 21B tokens of savings, across four realistic mixtures. For PPT research the design target shifts from imitating linguistic hierarchy to forcing long-range retrieval. Set's failure is a useful negative result: structure alone is not enough; locatable dependencies are what transfer. The phenomenon survives; the explanation does not.
Why OLMo3 fails at more than one scale is left open; a direct ablation was beyond the authors' compute after the 3B runs alone consumed about 1.07T tokens. The 7B results cover a single mixture and stop at 75.5B tokens (11 tokens per parameter vs 33 at 3B), so capacity saturation and insufficient budget cannot be separated, and the 7B runs use a lower learning rate, an extra variable. The long-range-retrieval account is correlational: stable verbatim-retrieval gains, retrieval-demanding PPT tasks, and context-dependent benchmarks all point the same way, but no experiment turns retrieval off. Most configurations are single runs; only 3B Marin gets three random seeds, with the rest of the conclusions resting on cross-checkpoint stability.