2026-10-24
Stanford injected strings into Pythia pretraining. One exposure peaks at 12±3.7 tokens on 6.9B; later checkpoints memorize more, and unlearning still leaks 10 to 15 tokens.
Verbatim memorization is a model emitting a long stretch of training text token for token. That is a different failure from storing a fact such as a capital city. Copyright and privacy exposure sit in the literal reproduction.
A natural hope is that a few neurons or one attention head are dedicated to reciting those strings, and that editing them would delete the text while leaving the language model intact. Observational studies do tie memorization to repetition and to model size. They cannot separate "this string was frequent" from "the model would have produced this kind of sentence anyway."
The single-exposure story is weaker than it looks. On the deduplicated Pythia 6.9B, 94% of strings counted as memorized show up at least once per 5 million examples. The remaining 6% were labeled by hand: templated text 54%, spacing or punctuation variants 21%, inductive patterns such as counting sequences 17%, and compositions of frequent fragments 8%. Checkpoints from before those strings appear can already generate them. What looks like one-shot memorization is usually ordinary completion.
Jing Huang, Diyi Yang, and Christopher Potts at Stanford study this with sequence injection. Pythia is the EleutherAI family that releases both checkpoints and the training order. From one checkpoint they train a pair. The control continues on the original data. The treatment continues on the same data, with a set of novel sequences swapped in at a fixed interval. Optimizer state is matched, so the pair is a practical counterfactual: what would the next token have been if this string had never been seen?
They inject 100 sequences of 256 tokens, drawn from Wikipedia, BBC, GitHub in languages released after 2020, papers, and novel excerpts, all dated after the Pile cutoff at the end of 2020. Data Portraits (50-character hashes) and infini-gram confirm overlap with the Pile stays under 50 characters. A shuffled twin set keeps the vocabulary distribution and drops predictable structure.
The metric follows Carlini and colleagues. Prompts are substrings of 8, 16, 32, or 64 tokens; greedy decoding runs for 64 tokens; the score is the longest prefix that matches the source. A string counts as memorized when some prompt of at most 32 tokens yields at least 32 matching tokens. The main runs use Pythia 160M and 2.8B at the 1K, 10K, and 80K-step checkpoints, with an injection every 50K or every 10K examples, and injected text under 0.04% of the data. The 160M models are measured after 40 occurrences, the 2.8B models after 20. The 6.9B model is used for frequency counts, the single-exposure run, and causal measurements on the pretrained checkpoint, not for the continued-training grid.
The mechanism tests are interchange interventions, also called activation patching. While the model is about to predict the next memorized token, the residual stream at one prefix position and one layer is replaced with the residual from a random Pile sequence at the same site. The residual stream is the hidden state after attention and MLP outputs have been added back. If the prediction changes, that site is causally involved. If it does not, the token is coming from ordinary language-model decoding. A cross-model variant writes internal states from the treatment model into the control, which has never seen the string, and asks how far the control's own attention and MLP can carry the memorized continuation.
One exposure does not store an arbitrary passage. The 2.8B and 6.9B checkpoints at 80K steps see the target only in the first batch, then train for 200 steps, with measurements every 10 steps, averaged over 16 sequences. Memorized length falls as batch size grows, and it peaks about 25 to 100 steps after the exposure, which the paper attributes to optimizer momentum. At the peak, Pythia 6.9B with batch size 128 retains 12±3.7 tokens of the original string. Shuffled strings stay at 5±1 tokens across model sizes and batch sizes. Real pretraining uses batch 1024 and a context of 2048; this run uses a window of 224. Under that gap, models smaller than 6.9B are extremely unlikely to verbatim-memorize an arbitrary sequence in a single pretraining step.
Later checkpoints memorize longer strings. In the observational sample, the median frequency of memorized sequences drops from 4×10⁻⁵ at 160M to 1×10⁻⁵ at 2.8B and 9×10⁻⁶ at 6.9B: larger models memorize rarer strings. Injection separates quality from size. At one injection per 50K examples, later checkpoints of the same size memorize longer prefixes, and the 2.8B model after only 20 occurrences already beats the 160M model after 40. In-distribution strings are memorized more readily than shuffled ones, except at the 2.8B 10K-step checkpoint. Shuffled strings still get longer at later checkpoints, just not as long. Out-of-domain text is not preferred merely because it contains rare tokens that could act as lookup keys. At one injection per 10K examples the gap narrows on 160M, and on 2.8B shuffled strings remain harder except at the 10K checkpoint.
Not every verbatim token depends on the trigger prefix. Many dependencies look like ordinary syntax: a verb governs its object, a closing parenthesis tracks the opening one. On strings without that structure, the dependencies land on relatively rare trigger tokens. Where a trigger dependency exists, it sits in middle-layer states. On the 160M model at the 80K checkpoint, in the setting the paper states as one injection per 10 million examples, layer 4 still shows about one causal link on average (threshold 0.1; 50 injected sequences; 100 random Pile texts each). No single representation of the prefix encodes the whole memorized string. The same middle-layer pattern shows up on 50 memorized sequences from the pretrained 6.9B model.
General language modeling accounts for a real share of the output. About a quarter of the tokens scored as memorized in the treatment model are already predicted by the control, which never saw the string; those tokens are dropped from the later analysis. On the rest, patching the treatment model's last-layer residual into the control succeeds on 85% of inputs, so the two models' representations are interchangeable enough for the test to mean something. In most attention and MLP layers, more than half of the interventions that produce the next memorized token still work when that layer's weights come from the control. The share falls to about 30% in MLP layers 1 and 3 and attention layers 1 and 4, so those layers differ more across the pair and are likelier to hold string-specific information. They still do not form a cleanly removable memory. Asking an intervention to produce 4 memorized tokens in a row drives the success rate to about zero. No single component carries the passage.
Freezing attention and training only the MLP shows which block is required. In Pythia the MLP is about 35% of parameters and attention heads about 17%. At one injection per 50K examples, neither MLP-only nor attention-only training memorizes the strings. At one per 10K examples, frozen attention still allows an average memorized length of 40 tokens, about 80% of the fully trainable model. Storage depends more on the MLP. Attention patterns learned for ordinary language modeling can be reused when the string is repeated often enough.
Unlearning fails once the prefix moves. The stress test uses GPT-Neo 125M, a separate Pile model, under the Stoehr et al. protocol: a 50-token trigger, a 50-token continuation, 90 sequences. Gradient ascent and sparse fine-tuning run for 10 steps at learning rate 1×10⁻⁵; the sparse run updates only the 0.1% of weights with the largest gradient. Neuron pruning removes 0.1% of neurons with an L1 penalty of 1000; a higher penalty collapses generation. Position perturbations slide the window into the memorized span. Semantic perturbations swap in similar words or digit strings. Exact-match length in tokens:
| Method | Original prompt | Position perturbation | Semantic perturbation |
| Gradient ascent | 19±18 | 35±15 | 31±21 |
| Sparse fine-tuning | 23±20 | 36±16 | 34±21 |
| Neuron pruning | 4±4 | 14±8 | 11±10 |
Perturbed prompts extract 10 to 15 more tokens on average. For gradient ascent and sparse fine-tuning, the share of sequences that come out in full (all 50 tokens) rises from 22% to 56%. Pruning moves less under the perturbations, and it often degenerates into repetition on the shifted prefixes. On the 90 sequences, self-accuracy drops by 0.248 and neg-accuracy by 0.094. The original prompt looks forgotten. A nearby prompt still yields the text.
Editing a handful of neurons is not, on this evidence, a way to lift copyrighted text out of a model. The states that trigger a memorized continuation are distributed, abstract, and shared with ordinary decoding. Stronger checkpoints memorize more from the same number of repetitions, so a "safe" duplicate count measured on a small model does not transfer upward. Deduplication budgets should be set against the stronger model.
Evaluation has a cheap fix. Testing unlearning only on the training prefix marks leftover memorization as deleted. Window shifts and near-synonym swaps add more than 10 extractable tokens.
There is no new unlearning algorithm here. The usable piece is the injection pair itself, on models that release data order and checkpoints, because it separates "the model saw this" from "the model would have written this anyway." Code and data are released as explanare/verbatim-memorization.
Injection training stops at 2.8B. The paper is explicit that fully open weights, data order, and hyperparameters were mostly the Pythia suite at the time. The 6.9B model was not continued-pretrained in this framework, and later open families such as OLMo are unchecked. The continued-training window is 256 tokens, with batch 128 at 160M and batch 40 at 2.8B, against a pretraining setup of context 2048 and batch 1024. The single-exposure run uses about twice the learning rate of the 80K-step schedule and a window of 224, both of which make one-shot memorization easier. The direction of the bias is clear, and the paper never reruns the measurement at the real context length.
Unlearning is measured on GPT-Neo 125M, not on the injected Pythia models, so the two "memories" are not the same strings. Standard deviations in the table are on the order of the means. The jump from 22% to 56% needs the length histograms in the appendix; it is not a stable effect size by itself. Semantic swaps come from ChatGPT, with no separate check that the substitutes are actually close in meaning.
The verbatim metric misses memory of another kind. The paper notes that models can store abstract states such as syntactic templates, and a token-match score will undercount that. At the 2.8B 10K checkpoint, shuffled strings are memorized longer than the originals, and the paper flags the exception without explaining it. The causal-intervention setup is described as one injection per 10 million examples, next to main-grid frequencies of 10K and 50K, with no reason given for the change.