Scaling Domain Data Repetition in LLM Pretraining
Jingwei Li, Xinran Gu, Rui Dai, Xintong Hao, Chengyin Xu, Yan Wu, Shuran Zheng, Jingzhao Zhang
cs.AI
2026-08-14
ByteDance Seed and Tsinghua redo domain-data repetition experiments under a fixed tokens-per-parameter scaling regime: the optimal repetition count correlates with domain validation loss at Pearson -0.944, rises mildly with model size, and counts tuned on small proxy models transfer conservatively to larger ones.
High-quality domain data (code, math, wiki, medical) is far harder to scale than generic web text, so as token budgets grow, its share of the training mixture shrinks. Repetition is the obvious fix, but a decade of prior work concluded that larger models overfit repeated data faster, so repeat less at scale. That conclusion carries a hidden premise: cross-scale comparisons held the training-data size fixed. Under the Chinchilla-style regime people actually use, where token budgets grow with model size at a constant tokens-per-parameter ratio, the paper shows the conclusion flips: at fixed TPP, the optimal repetition count rises with model size.
Each model size N gets a budget DN = TPP·N with TPP held above 100, and every run at that size consumes the same tokens. For four domains (Code, Math, Wiki, Medical), the authors sweep unique-data fraction (1/40, 1/20, 1/10) and repetition count (1-7), filling the remaining budget with non-repeated web data. Only one domain is repeated per run. Optimization uses Muon with power-law-scaled learning rates and batch sizes. For each configuration, final validation loss is fit with a quadratic over repetition counts, and the fitted minimum serves as the estimated optimum, smoothing out the discrete grid. A one-hot linear-regression theory decomposes risk into knowledge-acquisition error and noise-fitting error: repetition reduces the first and inflates the second, which explains both the negative loss correlation and the opposite size trends under fixed-D versus fixed-TPP scaling.
| Factor | Pearson correlation with optimal repetition count |
| Domain minimum validation loss | -0.944 |
| Model size | +0.400 |
| Unique-data fraction | +0.018 |
Domains differ sharply. Math tolerates the most repetition, with an optimum of 5-6; Code sits at 4-5; Wiki and Medical at 3-4. Substituting repeated tokens for unique tokens at a fixed domain budget, Math from 1 to 4 repetitions costs only a small loss increase, while Wiki degrades clearly beyond 2. Replacing unique tokens with repeated ones leaves OOD validation loss on ArXiv and News essentially flat when total domain share is held fixed; the damage stays in-domain. Learning-rate schedules matter on their own: earlier decay in a WSD schedule reduces the tolerable repetition count, and a constant learning rate tolerates the most. Repeated samples seen during low-learning-rate phases get memorized more tightly.
The paper turns into a direct recipe: sweep repetition counts on a small proxy model at the same TPP as the target, and the result is a safe bound for the larger model, because a count that avoids overfitting on the small model also avoids it at scale under the same TPP. The proxy sweep does not need to replicate the target's unique-data fraction, since the optimum barely moves with it. There is no universal exchange rate between repeated and unique data; estimate it per domain, with math corpora standing up to 4 repeats and wiki degrading past 2.
Model sizes and full training configurations are not disclosed, so reproduction is hard. Each run repeats exactly one domain, while real pretraining repeats several simultaneously; the authors list multi-domain interaction as future work. The theory is a one-hot linear regression, so its guidance for real Transformers is qualitative. The OOD conclusion depends on holding web-data volume fixed; if repetition crowds out web tokens, it may not hold. The repetition grid stops at 7, and Math's optimum of 5-6 sits near the edge, so the true optimum could be higher.