Stanford study: API benchmark scores run 3.4 points higher than chatbot interfaces
StanfordAILab · x · 2026-09-17
A Stanford team (Jennifer Wang, Joachim Baumann, Daniel E. Ho, Sanmi Koyejo; EMNLP 2026) audited 7 systems across ChatGPT, Claude, and Gemini on 9 benchmarks and found API evaluations don't transfer to chatbot interfaces.
Key findings:
- API evaluations score 3.4 points higher in accuracy and 2.1 points higher in test-retest agreement than interface evaluations.
- For ChatGPT, the API-vs-interface gap rivals the API-only gap between GPT 5.3 and GPT 5.4 — switching access surfaces can degrade performance as much as a full model generation downgrade.
- Varying system prompts, sampling parameters, and reasoning settings failed to reliably reproduce interface behavior; whatever sits between endpoint and deployed product can't be reconstructed by auditors.
Implication: eval reports should state the access surface, and audits via API may certify a different system than the one most people actually use.
Related event: Stanford Study: API Audit Results Don't Transfer to Chat Interfaces(4 posts)→
More from Safety
- Anthropic and Google Find Critical UEFI Firmware Flaw Affecting ~90% of Non-Mac Machines — gleech · 2026-11-02
- JPMorgan, Schwab and Amazon block AI shopping agents; Schwab says it stops dozens of attempts weekly — emmanuelvivier · 2026-11-02
- OpenAI tightens protections for under-18 users, first community safety report due early 2027 — emmanuelvivier · 2026-11-02
- Malicious npm cluster posing as AI SDKs harvests AWS credentials, dating back to October — cyb3rops · 2026-11-02
- Self-Hosting LLMs at a Law Firm: Infra and Data Governance Harder Than the Model — EducationalGroup7245 · 2026-11-02
- TAWA GUARD Launches Pre-Model Input Checks to Block Hidden Prompt Injections in Agent Inputs — TawaGuard · 2026-11-02