Emergent and Predictable Memorization in Large Language Models
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, Edward Raff
cs.CL
2023-04-22
On Pythia, a trained 70M model recalls just 19.7% of what 12B memorizes (precision 0.956). The 12B's early checkpoint hits 0.50 recall; ~0.9 comes only late in training.
A trained language model can emit long stretches of its training data verbatim. When those stretches contain personal data, whether the model can ship depends on whether it will reproduce them. Deduplication lowers average memorization across a corpus. It does not lock down one specific string. Differential privacy hurts pretraining quality, so teams training large models rarely use it. An inference-time filter can block verbatim copying with a reported 100% success rate; the same line of work notes that the filter is easy to bypass and does not stop the model from learning the string during training.
A full run at billions of parameters is expensive. Cost is estimated as C = 6 × parameters × tokens. The engineer wants to know, before that budget is spent, whether a small set of sensitive sequences will be generated verbatim by the finished model. A false alarm discards a cheap run that could have been continued. A miss spends the full budget on a model that can leak those sequences. Recall is the metric that matches the cost. Extra alarms are tolerable. Misses are not.
Memorization is k-extractible: a training string the model reproduces exactly when prompted with the k tokens that precede it. Here k is 32. Each training sequence is 2048 tokens drawn from shuffled documents. Only the first 64 tokens are scored, 32 as the prompt and 32 as the gold continuation. The memorization score is the fraction of those 32 continuation tokens that greedy decoding matches. A score of 1 counts as memorized. One forward pass costs about a third of a gradient step. Scoring full sequences at seven checkpoints would exceed the cost of training, so each sequence is scored on its first 64 tokens only.
The cheap run is a classifier. If it memorized a sequence, the prediction is that the final model will too. One setup uses a fully trained smaller model to predict a fully trained larger model. The other uses an intermediate checkpoint to predict the fully trained model of the same size. The suite is Pythia, 70M through 12B, all trained on the Pile in one public data order. GPT-Neo changes the data slightly across sizes. Some BLOOM models publish only three partial checkpoints. Cerebras-GPT publishes none. Neither can support the comparison. The seven checkpoints have seen about 23, 44, 65, 85, 105, 126, and 146 million sequences.
The Pythia technical report found that a sequence's position in the data does not change whether it is memorized. That suggested moving sensitive strings to the front of training and treating an early checkpoint as a stop signal.
Fully trained smaller models predict Pythia-12B with high precision and low recall.
| Predictor | Precision | Recall |
| Pythia-70M | 0.956 | 0.197 |
| Pythia-160M | 0.948 | 0.289 |
| Pythia-410M | 0.940 | 0.401 |
| Pythia-1.0B | 0.931 | 0.512 |
| Pythia-1.4B | 0.926 | 0.554 |
| Pythia-2.8B | 0.909 | 0.658 |
| Pythia-6.9B | 0.884 | 0.795 |
Of sequences memorized by the 70M model, 95.6% were also memorized by the 12B model, and those sequences are only 19.7% of what the 12B model memorized. A 1.4B model, about an order of magnitude smaller, reaches 0.554 recall. As the size gap grows, overlap in which sequences are memorized drops quickly. Failure to memorize a string in the small model does not establish that the large model will fail too.
Checkpoints inside one run show the same split. The comparison uses the first 23 million sequences, the data the earliest checkpoint has seen.
| Sequences seen | 12B precision | 12B recall | 6.9B recall |
| 23M | 0.918 | 0.500 | 0.513 |
| 44M | 0.915 | 0.575 | 0.587 |
| 65M | 0.913 | 0.641 | 0.658 |
| 85M | 0.911 | 0.711 | 0.721 |
| 105M | 0.916 | 0.809 | 0.816 |
| 126M | 0.943 | 0.916 | 0.918 |
At 23M sequences the 12B checkpoint has 0.500 recall, below the fully trained 6.9B model at 0.795. Precision on the 12B run stays near 0.91 until the last checkpoint, where it rises to 0.943. Recall reaches 0.916 only at 126M sequences. Putting sensitive text first does not buy a cheap abort.
Against compute, the curves are not the log-linear shape common in scaling-law work. Precision is the stranger plot: partially trained 12B points do not follow the smaller models. For recall, a larger model wins at fixed compute in the cheap regime. Near 1% of the full training budget, parameter count stops mattering among equal-compute runs. Near 10%, the smallest model at that budget is the best predictor. When the trial costs at least about 1% of the final run, the better spend at fixed compute is a smaller model, with no more tokens than the final run, and with sensitive sequences placed at the front of its data.
The cheap signal confirms that a sequence will be memorized. Precision from smaller models onto the 12B model runs from 0.884 at 6.9B to 0.956 at 70M. If the cheap run copies all 32 continuation tokens, the large run usually does too. The guarantee engineers need is that a sequence will not be memorized. That shows up only after a large share of the target training compute, as recall moves from about 0.5 early to above 0.9 late.
A 70M trial cannot clear a sensitive subset. If the spare budget is within one or two orders of magnitude of the real run and recall is still short, change the data or cancel the run. Some sequences memorized by the large model cannot be forecast from smaller models. The paper calls this emergent memorization. Nothing here tests models past 12B.
k-extractible matches a privacy threat: recovering training text from a short prompt. Whether it is the right probe for how memorization builds during training is open, and the memorization score may fit that question better. Every result is on Pythia, with a replication on the deduplicated Pythia models. No other public suite has the same data order and checkpoint grid, so other hyperparameters, corpora, and architectures are untested. The analysis never reads the text. It only asks whether a cheaper model memorized the same span, so it cannot say which kinds of sentences are easier to memorize.
An earlier draft reported substantially higher checkpoint recall because of a bug in the analysis code. The corrected numbers are lower, and the conclusions were revised to match. The paper says threshold choices and deduplication do not change the main findings. Those ablation figures are not in the main text. The claim that position in the data does not affect memorization comes from the Pythia technical report and is not re-tested here. The threat model also assumes a sizable trial budget before the real run. A team that can afford only one training run cannot use the equal-compute recommendation.