Study: API-based audits of ChatGPT, Claude and Gemini don't transfer to chatbot UIs
kenziyuliu · x · 2026-09-16
Third-party auditors typically evaluate models via the API — but do those findings transfer to the chatbot interfaces users actually see?
A new EMNLP 2026 paper audited seven systems across ChatGPT, Claude, and Gemini and found they don't. Findings from API-level audits don't reliably hold at the chatbot UI layer, meaning API-only audits can misrepresent what real users encounter — auditors need to test the deployed interface too.
Related event: Stanford Paper: API Audits Don't Reflect Real Chatbot User Experience(2 posts)→
More from Safety
- Safety Researcher: OpenAI Agent Collaboration Is Trained, Not Emergent — vishalmisra · 2026-09-16
- Polymarket prices just 8% chance US-China reach AI pacing deal by 2026 — Polymarket · 2026-09-16
- Stanford EMNLP paper: API-level audits don't reflect what chatbot users actually get — StanfordAILab · 2026-09-16
- Redditor questions whether Viro AI's '100% clean energy' claims are greenwashing — EuphoricEye4964 · 2026-09-16
- Op-Ed: Why Researchers Fear Recursive Self-Improvement Could Run Out of Control — OmarUFlorez · 2026-09-16
- Six experts on whether AI could wipe out humanity: less doom, still no consensus — gedbarker · 2026-09-16