AISPA: User-Centric System Prompt Auditing for Large Language Model Applications
Xiangning Lin, Shenzhe Zhu, Shu Yang, Zhenyu Zhang, Haoqian Zhang, Yipeng Zhao, Chengxuan Qian, Tianwei Wang, Ziheng Zhang, Zhenlong Yuan, Dingcheng Wang, Juncheng Wu, Yuan Si, Jiaxin Liu, Baolong Bi, Robert Mahari, Tobin South, Dazza Greenwood, Zexue He, Rishi Bommasani, Sophia Kazinnik, Andreas Haupt, Samuele Marro, Erik Brynjolfsson, Alex Pentland, Jiaxin Pei
cs.AI, cs.CL, cs.CY, cs.HC
2026-07-31
AISPA audits system prompts of 88 AI products across eight user-rights dimensions: 98.9% have protective instructions, only 24% cover all eight, 38.6% still contain anti-user ones.
System prompts are the hidden instructions developers write to govern a model: how it identifies itself, how it handles privacy, what it does with dangerous requests. They are the factory settings of nearly every commercial AI product, yet almost never disclosed. That leaves a gap. Users and regulators cannot tell what a product is being told to do under the hood.
Most AI safety research looks the other way, at how to stop outside attackers from manipulating the model with injections or jailbreaks. The AISPA team, largely from Stanford with co-authors across MIT, CMU, and Oxford, turns 180 degrees: instead of protecting the system from outsiders, they check whether the system itself protects users.
AISPA is an eight-dimension audit taxonomy, each dimension traceable to specific articles of the Universal Declaration of Human Rights: identity transparency (D1), truthfulness (D2), privacy (D3), tool safety (D4), user agency and manipulation prevention (D5), unsafe-request handling (D6), harm prevention (D7), and fairness and neutrality (D8). Every instruction is labeled protective (+1) or problematic (-1). The audit is human-in-the-loop: Claude-4.6-Opus pre-annotates, six annotators screen independently, and three experts adjudicate. Pairwise inter-annotator agreement reached 0.933.
The corpus comes from six GitHub repositories of leaked or publicly disclosed prompts, covering 88 products and yielding 1,818 unique instructions (2,420 instruction-dimension entries). Four headline findings: 98.9% of products (87/88) carry at least one protective instruction; only 23.9% (21/88) cover all eight dimensions; 38.6% (34/88) still contain at least one problematic instruction; and average prompt length grew from about 9K to over 30K characters between 2024 and 2025.
Problematic instructions have not gone away. Their prevalence peaked at 67% of products in 2025 Q1, fell to 19% in Q3, and rose again to 29% in Q4. By organization, Anthropic leads on both ends, averaging 62.3 protective and 0.1 problematic instructions per product; Venice is the only product whose problematic count (3.0) exceeds its protective count (2.0). One telling asymmetry: the most widely covered dimensions, truthfulness and user agency, are also the most frequently violated, which means writing a rule does not guarantee following it.
This is one of the few efforts to treat system prompts as a quantifiable audit target. For practitioners it is a ready checklist: where does your product have gaps, and where do its hidden instructions conflict with user interests? For regulators and platform governance it offers a third-party lens that does not depend on vendor goodwill.
The authors flag two issues. First, every prompt came from leaked or public repositories, so the team cannot confirm they match the current production versions (cross-repository checks showed high overlap). Second, there is selection bias: products whose prompts are easy to extract are over-represented. An unaddressed concern is that several co-authors are affiliated with Anthropic, which ranks first in the audit. The ranking reflects leaked snapshots, not official current prompts, and readers should weigh that.