EMNLP 2026 paper: API-based audits of seven chatbots don't transfer to chat interfaces
stanfordnlp · x · 2026-10-26
Research presented at EMNLP 2026 (Oct 26, Hall H, Poster Session D) shows that third-party audits conducted via API don't transfer to chatbot interfaces.
The team audited seven systems across ChatGPT, Claude, and Gemini and found inconsistent results between API and chat UI channels, suggesting API-based safety audits may misrepresent the end-user experience.
The author also invites discussion of related work on ecologically valid evals and AI's impact on the future of work.
More from Safety
- Open weights' rise may end as US and China restrict cyber-capable AI models — GunmetalZen · 2026-10-26
- Claude Code audit finds 11,400 "vanishing" Instagram photos still cached on iPhone — TheMoonMidas · 2026-10-26
- User gets ominous Gemini policy-violation warning from Google with zero explanation — kingslyj · 2026-10-26
- Dev warns: without a hardware security key like YubiKey, you're heading for pain — KevinNaughtonJr · 2026-10-26
- voooooogel: AI will purge 50 years of tech debt and finish 90s cyberidealism — voooooogel · 2026-10-26
- Brave engineer on MCP tool poisoning: pre-execution model review and a 4GB local guard model — gnukeith · 2026-10-26