Stanford EMNLP paper: API-level audits don't reflect what chatbot users actually get
StanfordAILab · x · 2026-09-16
Stanford AI Lab is promoting a new EMNLP paper led by Jenn Wang arguing that auditing a model via the API doesn't tell you what users actually experience in the chatbot. The authors use this as timely evidence that the third-party access debate should center on systems, not just models — the same model wrapped in different system configurations can behave very differently, so audits limited to the model layer miss real-world behavior.
Related event: Stanford Study: API Audit Results Don't Transfer to Chat Interfaces(4 posts)→
More from Safety
- Cryptographers push back: claims ECC is likely to break are not consensus — jedisct1 · 2026-11-02
- UK's 'AI sovereignty' push undermined by its own energy and copyright policy — NinaDSchick · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- Cambricon co-founder urges shared US-China AI ethics standards for leading models — pstAsiatech · 2026-11-02
- Vitalik's "post-AI cryptography": pseudonyms are dead, per-action anonymity is in — devanshmehta · 2026-11-02
- 6 prompts to see what ChatGPT quietly knows about you — and wipe it — Aiden_Tech_Ai · 2026-11-02