Stanford post unpacks "palimpsestic" test: lightweight way to trace LLM provenance and theft
ChrisGPotts · x · 2026-10-23
Christopher Potts's blog post unpacks Kuditipudi, Huang, Zhu et al. (2025)'s provenance test: correlate M1's training data order with the logprobs M2 assigns to those examples to determine whether M1 is a derivative of M2. The test is remarkably lightweight — no full data order needed, robust to fine-tuning/model souping of M2, logprobs can be estimated from text samples or even M2's own generations, and M1 and its data need no modification. Potts also reports progress on predicting memorization: just three well-understood variables (Frequency, Recency, Hardness) now accurately predict verbatim memorization, based on Jing Huang's experiments with the Hubble model suite, answering an open problem from "Don't just Fix it in Post."
More from Safety
- 'Black October' security findings: AI agents make exploiting old flaws cost pennies — LexSokolin · 2026-10-24
- Google says it might use your Gmail and Photos data to target ads — The Verge AI · 2026-10-24
- IBM X-Force's Vibe Fuzzing Talk: LLM-Guided Fuzzing with Graph Inputs and Program State — dyn___ · 2026-10-24
- Greece passes nationwide social media ban for children under 15 — Polymarket · 2026-10-24
- AI Finds Software Flaws Faster Than Fix: 45,207 CVEs Logged by Late July — rohanpaul_ai · 2026-10-24
- Two OpenAI Safety Researchers Fired Over External Communications Speak Out — kevinroose · 2026-10-24