Domain-Aware Scaling Laws Uncover Data Synergy
Kimia Hamidieh, Lester Mackey, David Alvarez-Melis
cs.LG, cs.CL
2026-07-13
Fitting scaling laws to 52 open LLMs quantifies data synergy: code with math or science is the strongest pair, and trained 30M/150M models confirm the predicted optimal mixtures win.
The Chinchilla line of scaling laws, which describe how pretraining loss falls as a power law in parameters and tokens, treats the dataset as a single undifferentiated token count. Every token is assumed equal. Real pretraining corpora are not: they are mixtures of web text, books, code, math, and so on, and practitioners have long known these domains interact. Adding code tends to lift math reasoning; math and code together often beat either alone; some pairings actively interfere and depress scores. What was missing was a way to formalize these effects and, just as important, separate them from the plain benefit of having more data. This paper, from MIT and Microsoft Research, fills that gap.
Synergy is split into two layers. The first is domain-to-benchmark synergy: a domain changes the rate at which data drives down loss on a given benchmark. Concretely, the Chinchilla data exponent beta gets a per-domain, per-benchmark modifier gamma, where positive gamma means that domain reduces loss faster than the baseline rate and negative gamma means interference. The second layer is second-order synergy between domains: a pairwise "bonus tokens" term that fires only when both domains are present, capped by the scarcer of the two (a softmin, so the bonus is zero if either domain is absent).
Crucially, the authors do not train a controlled fleet of models. They exploit natural variation across 52 open-weight models from 70M to 20B parameters spanning six families (GPT-Neo, Pythia, DataDecide, OLMo, OpenLLaMA, RedPajama-INCITE), map each model's sources to eight domains, and evaluate on 11 benchmarks. The 30 DataDecide models carry controlled ablations (no-code, no-math-code, no-flan, no-reddit) that are especially informative for synergy.
The fitting is restrained: Huber loss for robustness, L1/L2 regularization to keep gamma sparse, 5-fold cross-validation, and 50 bootstrap resamples for 90% intervals; only entries whose interval excludes zero count as meaningful.
The strongest first-order synergies sit on the code and math benchmarks. On HumanEval, gammacode = +0.47 and gammamath = +1.34; on MBPP, gammacode = +0.59 and gammamath = +1.44. The most reliable interference comes from books and encyclopedia dragging down code benchmarks. Strip code from the dolma1.7 mixture and HumanEval's data exponent beta falls from 0.45 to 0.18.
For prediction accuracy, the first-order model reaches a median cross-validated R-squared of 0.906 across benchmarks. The token-count-only Chinchilla baseline manages only R-squared = 0.41 on HumanEval and 0.20 on MBPP; the first-order model hits 0.92 and 0.88.
| Model | HumanEval R-squared | MBPP R-squared |
| Chinchilla baseline | 0.41 | 0.20 |
| First-order synergy | 0.92 | 0.88 |
The top second-order pairs are code x science (+2.55) and code x math (+2.40); the clearest negative is math x books, two domains with little shared structure that dilute each other. The second-order model reaches a median R-squared of 0.912 and edges out the first-order model on GSM8K, IFEval, and TriviaQA.
The most telling test of actionability is Section 4.3. The authors use the estimated synergies to predict "optimal" and "anti-optimal" mixtures for HumanEval, GSM8K, and IFEval, then actually train 30M and 150M models on each. The rankings hold in every case. At 150M, the optimal HumanEval mixture beats the anti-optimal by 0.337 BPB, and GSM8K's advantage grows from 2.1% to 14.7% as scale increases.
| Task (150M, 5B tokens) | Optimal BPB | Anti-optimal BPB | Gap |
| HumanEval | 0.721 (-16.9%) | 1.058 (+21.9%) | 0.337 |
| GSM8K | 1.402 (-14.7%) | 1.591 (-3.2%) | 0.189 |
For anyone running pretraining, the payoff is an interpretable map of which domains reinforce each other and which cancel out, obtained without training a controlled model sweep. Code pairing with math and science is a finding that has surfaced repeatedly, and now has a quantitative scale attached; the negative pairings are directly actionable, since tokens spent on an interfering domain are tokens wasted for the target benchmark.
This is an incremental refinement of scaling laws, not a replacement: it bolts composition and synergy terms onto the Chinchilla data term. It is best read as a reference line for data curation and mixture design, not a substitute for real training experiments.
The authors are upfront about these. The estimates are observational, so the gamma values are associations rather than causes; they rest on mixture metadata that may be noisy or incomplete, and the observed mixture vectors are effectively low-dimensional, so gamma is a sensitivity along observed axes rather than a fully causal synergy. The controlled validation only reaches 30M and 150M, so behavior at larger scale is unverified. Whether the same pairs stay synergistic under supervised fine-tuning or reinforcement learning is left open.
One open question beyond the paper: synergy estimated at pretraining may not transfer cleanly to post-training stages. The eight-domain split is also coarse, treating all of code as a single block with no further subdivision.