Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei
cs.CL
2020-05-29
OpenAI trains a 175B autoregressive LM and, with frozen weights, reaches 71.8 few-shot SuperGLUE, above fine-tuned BERT-Large at 69.0.
By 2020 the default NLP recipe was pretrain, then fine-tune. BERT and T5 had already made the architecture task-agnostic, but every new task still needed thousands to hundreds of thousands of labeled examples. That is expensive, and fine-tuning on a narrow set invites spurious correlations. People generally pick up a language task from an instruction or a handful of demonstrations.
GPT-2 had already tried in-context learning: write the instruction and a few examples into the prompt, leave the weights frozen. It scored 4% on Natural Questions. This paper scales the same decoder-only Transformer to 175 billion parameters and measures zero-shot, one-shot, and few-shot side by side. The bet is that scale itself turns "learn from a few examples" into a usable skill.
The backbone is GPT-2: pre-norm, reversible BPE, 2048-token context. The new piece is alternating dense attention and locally banded sparse attention, in the spirit of Sparse Transformer, so width and depth can both grow. Eight models are trained, from 125 million to 175 billion parameters. The flagship has 96 layers, dmodel 12288, 96 heads, and a 3.2 million token batch. Every size sees 300 billion tokens.
Filtered Common Crawl supplies about 410 billion tokens and 60% of the training mix. WebText2, two book corpora, and English Wikipedia fill the rest. Higher-quality sources are oversampled: Wikipedia is seen about 3.4 times, Common Crawl less than once. Document-level fuzzy dedup is applied, and the authors tried to strip downstream eval sets. A filter bug left residual overlap.
At evaluation the weights never move. Zero-shot gets a natural-language instruction. One-shot adds a single demonstration. Few-shot packs the context, typically 10 to 100 examples. Multiple-choice items are scored by conditional likelihood of each completion.
Larger models use in-context information more efficiently. The gap from zero-shot to few-shot widens with scale.
| Task | Setting | GPT-3 | Reference |
| SuperGLUE average | few-shot, K=32 | 71.8 | fine-tuned BERT-Large 69.0; fine-tuned SOTA 89.0 |
| LAMBADA accuracy | few-shot | 86.4% | prior SOTA 68.0% |
| TriviaQA | few-shot | 71.2% | open-domain fine-tuned RAG 68.0% |
| CoQA F1 | few-shot | 85.0 | fine-tuned SOTA 90.7 |
| 2-digit addition | few-shot | 100% | the 13B model is about half |
Closed-book few-shot TriviaQA beats retrieval-augmented RAG. Translation is lopsided: into English is strong, out of English is weaker, and En→Ro sits more than 10 BLEU behind then-current unsupervised NMT. CoQA is close to humans; RACE-high is 46.8%; QuAC lands 13 F1 below an early ELMo baseline.
Synthetic probes make the scale jump blunt. The 175B model scores 100% / 98.9% on 2-digit add/subtract and 80.4% on 3-digit add. The 13B model is near 50% on 2-digit add/subtract and under 10% on most other operations. Two-digit multiply is 29.2%. Humans spotting 200-word news as machine-written fall from 86% on a degraded control to 52% on GPT-3, near chance.
On ANLI, every model below 175B stays near the 33% random line even few-shot. WiC is 49.4%. Comparing two sentences for paraphrase or entailment is the wall this recipe does not climb.
This is the paper that turned "the prompt is the task" from a GPT-2 demo into a reproducible scaling result. Instruction following, tool use, and a lot of later product surface area sit on that observation: you do not need a fine-tuned checkpoint per task if you can write the spec and a few examples into context.
Two caveats for anyone using the number. Few-shot SuperGLUE just clears BERT-Large and is still far from the 89.0 fine-tuned SOTA, so fine-tuning was not obsolete. Arithmetic and letter-manipulation jump between 13B and 175B; those skills were not a smooth gift at every size.
Section 5 is unusually direct. Samples still repeat, drift over long passages, and contradict themselves. Commonsense physics is weak ("will cheese melt in a fridge"). The model is decoder-only, so cloze, look-back comparison, and long-passage-then-short-answer tasks are structurally disadvantaged. That is a plausible reading of the WiC and ANLI collapse.
Deeper limits are named too. Next-token loss weights every token equally. Tasks have to be jammed into a prediction problem. There is no visual or bodily grounding. Contamination filtering failed for some benchmarks including LAMBADA; the authors say the effect is small, and nobody else can re-run the filter. The 2048-token window caps how many shots you can show. At 2020 compute, the 175B run is effectively unreproducible.