Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre
cs.CL, cs.LG
2022-03-29
DeepMind trains 400+ models and finds compute should scale parameters and data equally. Its 70B Chinchilla matches Gopher's 280B compute yet hits 67.6% on MMLU, up 7.6 points.
In 2020, Kaplan et al. gave GPT-3 its scaling law: bigger models do better, following a clean power law. The field ran hard with one corollary: for every 10x more compute, grow parameters 5.5x and training tokens only 1.8x. Every flagship dense model since, GPT-3, Gopher, Jurassic, the 530B MT-NLG, scaled parameters from 175B to 530B while training on a near-constant 300B tokens. Models ballooned; the data diet barely moved.
The question here is concrete. Given a fixed compute budget, and you usually know up front how many accelerators you have and for how long, should the money go into a larger model or into more data? These models are typically trained only once, so estimating the right split before you start is what decides whether the spend was worth it.
The authors train 400+ models, from 70M to 16B parameters on 5B to 500B tokens, then estimate the optimal Nopt(C) and Dopt(C) with three independent methods:
All three agree: as compute grows, parameters and tokens should scale together at roughly 1:1. The fitted exponents a and b land between 0.46 and 0.54 (equal scaling is 0.5 each).
Why did Kaplan get the opposite answer? Two differences matter. Kaplan used one fixed token count and one learning-rate decay schedule for every model, which systematically understates how well models train on less data. And most of Kaplan's runs sat below 100M parameters, whereas most here are above 500M and include 16B points; the compute frontier curves slightly downward at the high end, and small-model fits miss it.
Following the recipe, the authors train Chinchilla: 70B parameters, 1.4 trillion tokens, at exactly Gopher's compute (280B, 300B tokens).
| Benchmark | Chinchilla (70B) | Gopher (280B) | Note |
| MMLU (5-shot) | 67.6% | 60.0% | +7.6 pts, beats the human forecast of 63.4% for June 2023 |
| BIG-bench (mean) | 65.1% | 54.4% | +10.7 pts, behind on only 4 of 62 tasks |
| TruthfulQA (0-shot) | 43.6% | 29.5% | +14.1 pts |
| RACE-m / RACE-h | 86.8% / 82.3% | 75.1% / 71.6% | reading comprehension, up 10+ pts each |
| LAMBADA | 77.4% | 74.5% | also above the 530B MT-NLG |
On every Pile subset, bits-per-byte is lower than Gopher; Natural Questions and TriviaQA closed-book results lead across the board. For the same money, a 4x smaller model wins on almost everything.
The projection matters more than any single score. On the compute-optimal recipe, a 175B GPT-3-class model should have trained on 3.7T tokens (not 300B), and Gopher's 280B on 5.9T. Reaching a 1-trillion-parameter model optimally needs roughly 21T tokens and 250x Gopher's compute. The conclusion is that nearly every large model of the era was over-parameterized and badly undertrained.
This paper ended the "just add parameters" era. After it, Llama, Falcon, and Mistral all moved to smaller models trained on far more data, and Chinchilla's 70B / 1.4T-token point became a de facto reference. For practitioners, the same budget now buys a smaller model that scores higher, and a smaller model means lower inference and fine-tuning cost, so both the deployment bar and per-call cost fall together.
It also shifted the field's attention from making models larger to making high-quality data more plentiful. Data, not parameters, became the scarce resource.
The authors name a few. There are only two large-scale comparable runs, Chinchilla and Gopher, with nothing validated at intermediate scales. The whole framework assumes a clean power-law frontier, but Nopt curves slightly downward at high compute, which means they may still be overestimating optimal model size. Every run trains for under one epoch, so the multi-epoch, data-repetition regime is untouched.
One engineering caveat stands out. Chinchilla sees 4x Gopher's data, so train/test leakage can inflate the language-modeling numbers even on a shared corpus; the authors acknowledge this and lean on MMLU and BIG-bench, where leakage is less of a concern. The exponents are also fit on a single data mixture (MassiveText) and a single evaluation suite, with no cross-check that they hold when either changes.