Best-fit scaling on ViT-B/16 gives c=-0.24; extrapolation prefers c=-0.40

Revisiting Neural Scaling Laws in Language and Vision

Ibrahim Alabdulmohsin, Behnam Neyshabur, Xiaohua Zhai

NeurIPS)

cs.LG, cs.AI

2022-09-13

On ViT-B/16's ImageNet 10-shot curve, extrapolation picks c=-0.40 against a best fit of -0.24, and M4 wins on over 70% of image tasks.

What problem this solves

A scaling law is a curve for how error falls as models or datasets get larger. The usual shape is a power law: error goes like x to the power c, and c is negative. A larger absolute value means a steeper return to scale.

The usual procedure fits that exponent on the learning curve already in hand, then uses it to forecast a larger scale. ViT-B/16 pretrained on JFT-300M, scored with ImageNet 10-shot error, shows why the two steps come apart. The fit uses only points up to about 500 million examples seen. In that window, M2 curves with exponents from about -0.24 to -0.40 lie almost on top of each other. The best fit selects c = -0.24. From 1 billion to 2 billion examples, log RMSE prefers c = -0.40.

The exponent with the best fit has not been checked as a forecast.

Method

Four function families map error εx to a scale x. Here x is usually dataset size, or the number of examples seen in training.

The small-x end should sit near random guessing, so an early plateau does not drag the exponent. At large x the excess error has to become a power law. For α > 0 the correction is on the order of α x^{2c}, and it shrinks as the curve enters that regime.

Parameters minimize a square-log loss, so a relative miss has similar weight at every scale. Predicting 1.0 where the truth is 0.99, or 0.1 where the truth is 0.099, both give a log RMSE near 1%. Least squares solves α, β, and c in closed form. The remaining parameters use gradient descent at learning rate 10^{-7}.

A classification problem with a known answer makes the gap visible. Points are drawn uniformly on the 100-dimensional unit sphere, and labels flip with probability 0.2, so the Bayes error is 0.2. Logistic regression plateaus, drops quickly, then enters a power law. At cutoff τ = 0, M2 returns c = -0.14 and M4 returns -0.57. At τ = 1000, once the points sit in the power-law regime, M2 moves to -0.48 and M4 stays at -0.49.

The main protocol sets τ to half the largest x on the curve. Only the first half is used for fitting. The second half is held out. The score is RMSE after taking logs of the predicted and actual error.

Results

Image classification has 72 tasks. Two sizes each of BiT, ViT, and MLP-Mixer are pretrained on JFT-300M, about 300 million images and 18,000 classes. Table 1 lists BiT/50/1 at 61 million parameters, BiT/101/3 at 494 million, ViT/S/16 at 32 million, ViT/B/16 at 110 million, MiX/B/16 at 73 million, and MiX/L/16 at 226 million. Downstream scores are 5-shot, 10-shot, and 25-shot on ImageNet, Birds-200, CIFAR-100, and Caltech-101. Few-shot error is the metric because upstream loss is a poor stand-in for downstream accuracy.

On an upstream subset of about 12 million images, training too long pulls downstream accuracy off its peak. Before that peak, examples seen in training behave like fresh samples. That count is called bootstrapped examples, and the image experiments use it as x. M4 has the lowest extrapolation error on more than 70% of tasks in this domain. Within a family, the larger model has the larger |c|, and M4's absolute values sit above the other three families.

SettingComparisonResult
ViT-B/16, fit up to 500M examplesBest M2 fitc = -0.24
Same curve, extrapolate to 1B–2BBest M2 log RMSEc = -0.40
72 image tasksM1, M2, M3M4 lowest on more than 70%
Five language modelsM2 and M4c near -1/3, andcfalls as models grow
dateunderstanding, 1-shotM1/M2 at 3.2×10^{-2}M3 at 4.7×10^{-3}, M4 at 1.5×10^{-2}
linguisticmappings, 2-shotOthers near 1.7×10^{-2}M4 at 9.2×10^{-3}

Translation uses five architectures: a 6-layer encoder with a 6-layer decoder, 28 encoder layers with 6 decoder layers, 6 with 28, a decoder-only model trained with a language-modeling loss, and a Transformer encoder with an LSTM decoder. Training uses Adafactor, batches of 500,000 tokens, and dropout 0.1. The metric is holdout log-perplexity. The fit stops at 256 million sentence pairs and is checked only at 512 million. M4's extrapolation RMSE is lower than the other three.

Language modeling uses decoder-only Transformers in the LaMDA family, five sizes from about 1.68×10^7 to 1.07×10^9 parameters, with validation loss rescaled to [0, 1]. M4 and M2 are best, M4 slightly ahead, with the exponent near -1/3.

On BIG-Bench the base model is a 262-million-parameter decoder-only network. The five tasks with the highest learnability are linguisticmappings, qawikidata, unitconversion, multdatawrangling, and dateunderstanding, each at 1-shot and 2-shot, scored with the benchmark's own multiple choice grade or exact string match. M3 and M4 are best and roughly tied. M1 and M2 are worse and close to each other. On qawikidata all four sit near 4×10^{-3}. The paper names noisy learning curves as one reason.

Why it matters

These exponents are what people would plug into a data budget, an early-stopping rule, or an architecture search. On this ViT curve, -0.24 is flatter than -0.40. Planning the next chunk of data from the best fit understates the return to scale.

M4 leaves the training run alone and changes only how an exponent is read off a curve. The early part of the curve may bend. The tail still has to be a power law, without first cutting the head off by hand. It wins on more than 70% of the image tasks. The smaller gap on language modeling follows the same design: those curves are closer to a pure power law, so the estimator falls back toward M2.

The released benchmark scores later estimators with the same extrapolation error.

Limitations

The forecast horizon is about 2×. The main split goes from half of the largest x to the end of the curve. Accuracy across a factor of two does not show that the same ranking would hold across a factor of ten.

Each translation model is checked at a single scale, and the curves have few points. BIG-Bench covers five tasks and two shot settings, and the curves are noisy. That is too thin to pick a function family for language tasks in general.

The image results depend on JFT-300M, which is not public. Bootstrapped examples are tied to the peak of downstream accuracy. Change the learning rate and the peak moves, so the x-axis moves with it.

In vision transfer, the larger model in a family has the larger |c|. In language modeling it runs the other way: |c| falls as models grow and stays near 1/3. The two patterns cannot be swapped into each other's budgets.

Sample-size planning, early stopping, and architecture search are not closed loops in this paper. The discussion says M4 could speed up neural architecture search later. That experiment is not run. A lower extrapolation RMSE does not yet mean the selected model is better, or that the budget is smaller. Whether the ranking survives a different loss or a different split is not compared in the main text.

Terms

Source

What people are saying

Related papers

All paper explainers