Three variables predict LLM verbatim memorization with AUC up to 0.94, Hubble experiments show
ChrisGPotts · x · 2026-10-23
Chris Potts reports progress on an open problem flagged in the "Don't just Fix it in Post" paper: predicting which sequences a model will memorize verbatim — without training the full model.
Using the Hubble model suite, a linear model with just three interpretable features predicts verbatim extraction with AUC-ROC of 0.82–0.94 across 1B/8B-param models trained on 100B/500B tokens:
- Frequency (number of copies in the dataset): 0.81–0.93 alone
- Recency (steps since last copy): 0.82–0.90 alone
- Hardness (NLL on models that never saw the string): chance, 0.51–0.55
Frequency and recency do most of the work, with direct implications for data curation and privacy risk assessment.
Related event: Stanford: Three Variables Predict LLM Verbatim Memorization(2 posts)→
More from Research
- Marin's 535B run passes 80% of pretraining, beating its scaling-ladder forecast — dlwh · 2026-10-24
- CREATE benchmark still unsaturated: GPT-6.1-Sol hits 23.8 on creative utility — gregd_nlp · 2026-10-24
- EMNLP 2026 keynotes: Pascale Fung on world models, Neubig on open research — mohitban47 · 2026-10-24
- HarmAmp benchmark shows LLMs amplify harm over multi-turn chats; TrajSafe monitor intervenes early — alan_ritter · 2026-10-24
- 30-month study of 26,811 students: AI homework use raised scores 18% but exam scores fell 20% — lmoroney · 2026-10-24
- ~6K-param router-logit detector cuts VLM safety errors by up to 92% — shangbinfeng · 2026-10-24