Hubble: a Model Suite to Advance the Study of LLM Memorization
Johnny Tian-Zheng Wei, Ameya Godbole, Mohammad Aflah Khan, Ryan Wang, Xiaoyuan Zhu, James Flemings, Nitya Kashyap, Krishna P. Gummadi, Willie Neiswanger, Robin Jia
cs.CL, cs.LG
2025-10-23
Hubble's 8B models memorize the same duplicates less on 500B tokens than on 100B; on 1B models, text inserted only in the first quarter of training is forgotten.
A copyrighted passage comes back verbatim, an email comes back out of a biography, or a benchmark score rises because the test items were in the pretraining mix. Those are memorization risks: the string sits in the weights and later shows up as lower loss or as a generated continuation.
Small models trained on synthetic text can isolate causes, and they are far from anything that gets shipped. Looking at an already-trained suite such as Pythia skips the training bill, then leaves a sentence's simplicity tangled with how often the web repeated it. How many duplicates a test item needs before it sticks is not identified in that setup. Hubble, from USC and MPI-SWS, turns the inserted text, the duplicate count, and the training phase into knobs on 1B and 8B models, with weights, data, and checkpoints released.
The base is Llama 3 with the OLMo tokenizer, about 50K tokens rather than Llama's 128K. Embeddings are untied from the output matrix so intermediate layers can be read out. The 8B model has 36 layers; Llama 3.1 8B has 32. Pretraining uses the DCLM baseline, past 500B tokens after decontamination. The 100B run is the first 100B tokens of that same shuffle.
Standard models see only that corpus. Perturbed models splice extra text between two documents, wrap it in EOS tokens, never split it across sequences, and place at most one insert in each 2048-token sequence. Three risk types go in. Copyright: passages from popular and unpopular Gutenberg books, Wikipedia events written after the DCLM cutoff, and one side of an MRPC or PAWS paraphrase pair. Privacy: templated YAGO biographies with nine attributes (name, nationality, birthday, email, UUID, and others), European Court of Human Rights cases, and PersonaChat dialogues with randomized usernames. Contamination: PopQA, WinoGrande as cloze and as multiple choice, MMLU, HellaSwag, PIQA, plus post-cutoff ELLie and MUNCH.
Each example is assigned 0, 1, 4, 16, 64, or 256 copies. After duplication that is 79.9M tokens in 818k sequences: 0.08% of the 100B corpus and 0.016% of the 500B corpus, well under the roughly 3% repetition level where general capability has been seen to drop. Decontamination removes 7,540 colliding documents, under 0.002%. The core release is a 2×2×2 of size, insert or not, and 100B or 500B. Six further 1B models on 100B tokens insert the same text only in the first half, the second half, or a single quarter of training. Single-domain controls and a paraphrase run rewritten by gpt-4.1-mini ship with them. Training took about 200k A100 GPU-hours.
The evals are marked as a lower bound. Loss is length-normalized log likelihood. Loss-based choice picks the lowest-loss candidate. Generative eval scores a continuation by exact match or word recall. Biography attacks run from a nearly complete prefix down to the name alone.
On the 8B models, the same duplicates are memorized less after 500B tokens than after 100B, on nearly every task. The copy count is fixed; the string is simply rarer in the longer corpus. At a fixed count the 8B also memorizes more than the 1B, and the signal shows up at fewer copies. Timing runs exist only for 1B models on 100B tokens. Text inserted solely in the first quarter is not memorized by the end of training, and intermediate checkpoints show it fading once exposure stops. Text inserted only in the last quarter is more extractable than in the ordinary perturbed run.
| Setup | Comparison | Result |
| 8B, same duplicate count | 100B vs 500B | Weaker memorization on 500B in every domain |
| 1B/100B, insertion window | First 25% only vs last 25% only | Early-only text is gone at the end; late-only text is more extractable than the ordinary perturbed run |
| 8B/100B, YAGO, richest side information | 16 duplicates | Accuracy near 100%; name-only attacks fall off sharply |
| 8B/100B, infer a PersonaChat username | 64 duplicates, 10-way | 34%; username-to-persona selection stays near chance |
| 8B/500B, unpopular Gutenberg, Loss MIA | 1 vs 64 duplicates | ROC AUC 0.539 vs 0.996 |
| 8B/500B, YAGO Loss / MMLU MinK%++ | 1 vs 16 duplicates | 0.538 vs 0.897; 0.58 vs 0.943 |
| 100B, one contaminated copy | Perturbed vs standard | Seen PopQA, HellaSwag, and PIQA items rise; unseen items barely move except a small PopQA gain, and WinoGrande falls |
Single-domain runs match the full perturbed model on the domain they contain, so mixing the three risks in one run does not scramble the curves. On Wikipedia passages, loss already separates seen from unseen at 4 duplicates for the 8B/100B model. K-eidetic memorization, verbatim extraction of a span from a prefix, shows up at 16. Popular and unpopular books are nearly indistinguishable at 1B; at 8B, generative extraction of popular books is only slightly higher. Occupation, email, and UUID are not remembered to the same degree. On the 8B paraphrase model, name-only attacks at high duplication beat the verbatim perturbed model, which means the stored object survives rewording.
One contaminated copy lifts accuracy on that copy for PopQA, HellaSwag, and PIQA, and does almost nothing for unseen items. When the exam format does not match the inserted format, WinoGrande can get worse as duplicates increase. Paraphrased MMLU items are not answered by the paraphrase model either. Membership inference here has none of the publication-date shortcut that undermined WikiMIA. At one duplicate, Loss, MinK%, MinK%++, and Zlib all land between 0.52 and 0.58 AUC. At 64 duplicates and above, Loss and MinK% approach or hit 1.0 on the three reported sets. MinK%++ is the strongest attack overall, yet at 256 duplicates on unpopular Gutenberg it reaches only 0.949, where Loss is already 1.0.
Unlearning is tested with RMU, representation rerouting, and SatImp on the 8B/500B perturbed model. The forget set is half of the 256-duplicate examples; the other half is a near-neighbor keep set. None of the three hits the target of matching the standard model on the forget set and the perturbed model everywhere else. All three drift toward the standard model and take the keep set and the test set with them. SatImp gets closer. Using the in-distribution keep set as the retain set, instead of WikiText, leaves the pattern in place.
Duplicate count and insertion time are known, and the rows to forget sit in the same distribution as the near-neighbors to keep. The standard models are a reference that never saw the inserts. TOFU and MUSE study unlearning after finetuning. WikiMIA labels membership by publication date, and that date leaks the label.
Two pipeline habits follow. Dilute sensitive text by training on more tokens, and place it early so later training can overwrite it. Deduplication still does a different job. For evaluation, a handful of leaked items raises the score on those items and does not teach the task. A detector that watches only the aggregate number will miss the split. General benchmarks stay in line with open models at the same scale, and the 0.08% insert does not degrade the perturbed runs. 500B tokens is about 22× the Chinchilla-optimal set for the 1B model and 3.7× for the 8B, roughly 1.6× the tokens Pythia saw, and still far from Llama 3 at 15T.
Forgetting-by-ordering and the single-domain check exist only for 1B models on 100B tokens. The paper does not show the same early-insert forgetting at 8B or at 500B, and the gap to a 15T run is too large to treat dilution and ordering as settled at frontier scale.
The inserts are cleaner than web text. They are never split across sequences, at most one lands in each sequence, and together they are 0.08% of the 100B corpus. Popular books are barely more memorized than unpopular ones. That is weak evidence on the data-density hypothesis; this corpus may simply not discuss those books enough for density to matter.
On the same Wikipedia passages, loss separates at 4 duplicates and verbatim extraction at 16. Swapping the metric changes the copyright reading, and every metric here is marked as a lower bound. The unlearning case study is a negative result: the three methods also erase neighbors. Dilution lowers memorization and, as the paper states, does not dispose of copyright or privacy liability. The 8-layer versus 32-layer runs were trained, and the main text does not report them. Paraphrases come from gpt-4.1-mini, and whether the rewrites added facts is unchecked.