Stanford Study: API Audit Results Don't Transfer to Chat Interfaces

A Stanford EMNLP 2026 paper auditing seven AI systems found that API-based evaluations score about 3.4 points higher on average than the same models' chat interfaces. The results suggest API audit findings do not transfer to user-facing chatbots or reflect real user experience.

2026-09-16 ~ 2026-09-17 · 4 related posts

• Study: API-based audits of ChatGPT, Claude and Gemini don't transfer to chatbot UIs — kenziyuliu · 2026-09-16
• Stanford EMNLP paper: API-level audits don't reflect what chatbot users actually get — StanfordAILab · 2026-09-16
• Stanford audit finds API benchmark scores don't transfer to chatbot interfaces — stanfordnlp · 2026-09-16
• Stanford study: API benchmark scores run 3.4 points higher than chatbot interfaces — StanfordAILab · 2026-09-17