A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage
Rui Xin, Niloofar Mireshghallah, Shuyue Stella Li, Michael Duan, Hyunwoo Kim, Yejin Choi, Yulia Tsvetkov, Sewoong Oh, Pang Wei Koh
cs.CR, cs.CL, cs.LG
2025-04-28
A re-identification attack recovers 74% of MedQA after Azure PII scrubbing; DP synthesis stops the leak but drops QA accuracy below a no-notes baseline.
When hospitals or labs share notes and chat logs, the default move is to strip names, addresses, and dates, or to fine-tune a generator and emit a "synthetic" corpus. Privacy is then scored with lexical checks: the PII detector is quiet, ROUGE no longer matches the original, and the release is treated as safe.
Unstructured text keeps identity in combinations of habits, history, and social detail. An adversary who knows that someone drinks at the same bar every week, or just lost a job, can still retrieve the sanitized record and read off age or substance use. Re-identification is standard practice for census tables. Text has not had an equivalent, rewrite-tolerant test.
The adversary sees the full sanitized release plus a few pieces of side information about a target, scraped from public sources or personal knowledge. The goal is to link that side information to a record and infer the rest.
Two stages.
A ROUGE-L linker and scorer serve as the lexical baseline. Utility on MedQA is multiple-choice accuracy of LLaMA 3.1 8B given the sanitized patient profile. On WildChat it is a normalized chi-squared distance over conversation categories. GPT-4o rates coherence on a 1-to-5 scale.
Sanitizers fall in two groups. Record-level: Azure AI PII, Staab et al.'s iterative GPT-4o anonymizer, Dou et al.'s self-disclosure span scrubber, and sanitize-then-paraphrase. Dataset-level: Yue et al.'s control-code GPT-2 fine-tune, with and without DP-SGD, ε in {3, 8, 16, 64, 512, 1024}.
On MedQA, raw text scores 0.15 semantic privacy and 0.69 QA accuracy. Azure PII reaches 0.73 lexical distance but only 0.26 semantic privacy, so 74% of the information remains, with accuracy still 0.67. Iterative anonymization and span sanitization reach 0.53 and 0.55 semantic privacy, accuracy 0.62. Non-DP synthesis sits at 0.52 / 0.61.
| Method | Semantic privacy (higher is safer) | MedQA accuracy |
| No sanitization | 0.15 | 0.69 |
| Azure AI PII | 0.26 | 0.67 |
| Iterative anonymization | 0.53 | 0.62 |
| Non-DP synthesis | 0.52 | 0.61 |
| DP synthesis, ε=1024 | 0.90 | 0.42 |
| DP synthesis, ε=3 | 0.92 | 0.41 |
| Drop the notes | n/a | 0.44 |
DP jumps semantic privacy to 0.90 at ε=1024 and 0.92 at ε=3, while accuracy falls to 0.42 and 0.41, below the 0.44 baseline that deletes the patient profile. GPT-4o coherence falls from 3.48 (non-DP synthesis) to 2.04 at ε=3, about a 36% quality drop.
WildChat repeats the Azure pattern: semantic privacy 0.26, category utility 0.96. Non-DP synthesis is much more private (0.83) and less useful (0.63). At ε=3, utility is -0.46, worse than a uniform category distribution.
Auxiliary claims alone recover about 20% of MedQA and 35% of WildChat. Non-DP sanitizers leak 45% to 74% once the sanitized set is in play, so most of the leakage is in the release. Three annotators labeled 580 claims (Fleiss κ=0.87); LLaMA 3 8B tracks the human mode at Spearman about 0.92 to 0.95.
A green PII report is not a release decision. Commercial scrubbers target names and dates and leave history of present illness, social history, and lab results intact. Synthesis without DP remains linkable. DP is the only method in this study that actually suppresses semantic leakage, and on a hard QA task it can be worse than not using the notes.
The contribution is a reproducible lower-bound attack, not a new sanitizer.
The study covers medical notes and chat logs only. DP numbers are tied to one DP-SGD fine-tune recipe and do not bound differential privacy as a theory. A high privacy score is not a certificate; occasional high attack success on DP text may be judge hallucination. Semantic similarity is the wrong lens for identifiers that die when a single digit changes. The side information is sampled from the original record, which is stronger than many real adversaries will have.