Countering Privacy Nihilism
Severin Engelmann, Helen Nissenbaum
cs.CY, cs.CR
2025-07-24
Engelmann and Nissenbaum argue that abandoning data categories because AI can infer “everything from everything” mistakes conceptual overfitting for unbounded inference.
Most privacy law works by sorting data: sensitive versus not, private versus public. GDPR Article 9, HIPAA, CCPA, and China's PIPL all take that path. A harder claim now circulates in AI: models can infer almost any attribute from almost any trace, "everything from everything" (EfE). If that is true, category-based protection collapses. Engelmann and Nissenbaum call the move of discarding data categories because one accepts EfE privacy nihilism.
They want to block two exits: treating AI inference as unbounded, which tells regulators to give up, and clinging to "data type only" rules that pretend inference is not happening.
This is a normative paper. No new model, no leaderboard. The argument has three moves.
Pin privacy nihilism to EfE. The target is not "AI infers," it is "inference is so powerful that category-based protection no longer makes sense." Then introduce conceptual overfitting: developers adopt norms of convenience that force complex constructs (political affiliation, sexual orientation, mental health) onto data that under-represent those constructs, or are irrelevant to them. Then bring in Nissenbaum's contextual integrity: legitimate information flow is specified by five parameters, not by data type alone.
Conceptual overfitting is walked through the pipeline.
Data collection follows the streetlight. Organizations dig where data is easy: social-media profiles, ImageNet, CommonCrawl. Kitchen-sink likes and friend counts are used to "discover" political orientation in users who never disclosed it. Patterns are treated as semantics without a prior conceptual commitment to the construct.
Ground-truth manufacturing is equally convenient. Supervised models need labels. Race and emotion labels often come from crowd workers mapping images onto a preset list, with majority vote collapsing disagreement into one "correct" meaning. Mental-health work often multiplies a questionnaire score by social-media traces. A widely cited study paid crowd workers to take the CES-D depression scale, then trained a random forest on Instagram color hues and claimed depression markers. Clinical diagnosis combines interview, history, exam, and scales; hue correlation does not carry the same diagnostic weight. Proxy hopping is worse: accelerometer traces infer walking, walking infers emotion, and each hop needs its own ground truth, so measurement error compounds.
Evaluation then rubber-stamps accuracy. Nobody can say whether 72% "politics from a face" is high enough. A cutoff turns a probability into a hard class, and in an overfitted model that cutoff is arbitrary. The same actors often build the model and report the score. SOTA contests on lab datasets make "it scores" look like "it infers."
There is no new experiment. The yield is a normative claim, not a table of accuracies.
EfE fails as a descriptive premise. Many of the most cited "sensitive from non-sensitive" demos conceptually overfit at collection, labeling, or evaluation. A high accuracy does not mean the model is inferring the construct itself. One cannot get "abandon data categories" from "AI looks unbounded."
The inference problem remains. Even if EfE is hype, amplified inference still hits type-only regulation: so-called public data can be assembled into information that should not flow. Data brokers, ClearviewAI, and some large-model training have used "we only collect public data" as cover.
The response is not to relabel public as private. That is a stopgap. Contextual integrity treats privacy as appropriate flow. A norm must specify subject, sender, recipient, data type, and transmission principle (consent, authorization, reciprocity, and so on). HIPAA already uses these levers: who may receive which health data, what they may do, under which conditions. Legitimacy is whether the flow serves the context's ends and values: care in medicine, learning in education.
Two hard lines, one for regulators and one for model builders.
Regulators should not declare category-based protection dead because of Kosinski-style "orientation from a face" results. Those results often have no conceptual accountability. Data categories should stay, but they cannot be the only knob. Who sends, who receives, why, and under what constraint is how inference gets governed.
For people who train models, fishing a sensitive construct out of kitchen-sink data will be read by press and law as a capability demo. This paper files it as an epistemic shortcut. If the claim is depression or political affiliation, write the construct first and show why the data bear on it. The accuracy number does not say what you inferred.
Conceptual overfitting is a diagnostic label, not a reproducible metric. The paper never says how much relevance counts as "conceptually enough," and it does not count how many EfE claims are overfit versus genuine inference. AlphaFold is the counterexample (fifty years of PDB labels), but the mix of theory-grounded scientific inference and theory-free social inference in current models is not measured. How to encode the five CI parameters into GDPR-style text is left for later, by the authors' own admission. One coauthor originated CI, so the framework choice is not neutral. The 2024 U.S. election and tech-oligarchy passage is loosely tied to the core argument.