Stanford audit finds API benchmark scores don't transfer to chatbot interfaces
stanfordnlp · x · 2026-09-16
Stanford NLP researchers (Jennifer Wang, Joachim Baumann, Daniel E. Ho, Sanmi Koyejo) audited ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy, challenging the assumption that API measurements reflect deployed behavior.
Key findings:
- API evaluations score on average 3.4 percentage points higher in accuracy and 2.1 points higher in test-retest agreement than the corresponding chatbot interfaces
- For ChatGPT, the API-vs-interface gap exceeds the API-only gap between GPT 5.3 and GPT 5.4 — switching access surfaces can degrade performance as much as downgrading a full model generation
- Tuning system prompts, sampling parameters, and reasoning settings shifts behavior in some cases but does not reliably close the gap
The paper calls this a "context-validity gap": API measurements don't necessarily generalize to deployed interfaces, complicating the use of benchmark scores for purchasing decisions, public trust, and policy. The authors also pitch Stanford as a credible third-party auditor.
Related event: Stanford Study: API Audit Results Don't Transfer to Chat Interfaces(4 posts)→
More from Research
- New benchmark shows frontier VLMs still can't read Braille despite 43M blind users — prajdabre · 2026-11-02
- Diffusion models emerge from associative memory: spurious states may signal generalization — DimaKrotov · 2026-11-02
- r/LocalLLaMA Proposal: Private Quality Benchmarks Over Endless tok/s Chasing — Express_Quail_1493 · 2026-11-02
- SIGGRAPH Asia 2026 heads to Kuala Lumpur Dec 1-4, technical papers program previewed — tomaszbednarz · 2026-11-02
- AI-generated phase transition simulation wows Reddit with order-parameter visualization — NedRadnad · 2026-11-02
- KAIST's QCP lets robot policies control execution quality in real time — kaist-ai · 2026-11-02