Study: API-based audits of ChatGPT, Claude and Gemini don't transfer to chatbot UIs
kenziyuliu · x · 2026-09-16
Third-party auditors typically evaluate models via the API — but do those findings transfer to the chatbot interfaces users actually see?
A new EMNLP 2026 paper audited seven systems across ChatGPT, Claude, and Gemini and found they don't. Findings from API-level audits don't reliably hold at the chatbot UI layer, meaning API-only audits can misrepresent what real users encounter — auditors need to test the deployed interface too.
Related event: Stanford Study: API Audit Results Don't Transfer to Chat Interfaces(4 posts)→
More from Safety
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- Cambricon co-founder urges shared US-China AI ethics standards for leading models — pstAsiatech · 2026-11-02
- Vitalik's "post-AI cryptography": pseudonyms are dead, per-action anonymity is in — devanshmehta · 2026-11-02
- 6 prompts to see what ChatGPT quietly knows about you — and wipe it — Aiden_Tech_Ai · 2026-11-02
- Insider: critical infrastructure firm flunked security audit, still delays basic fixes — thedealdirector · 2026-11-02
- Cyber attack on Swedish smart fridges causes mass food spoilage at supermarket — Polymarket · 2026-11-02