Stanford study: all 8 LLM judges rank their own medical answers higher, by 1.2 spots on average

TheTuringPost · x · 2026-11-19

A Stanford paper tested 8 models from OpenAI, Anthropic, Google and Qwen: each answered 620 real clinician questions, then all 8 anonymously ranked every answer.

The authors note many weekly leaderboards rely on a single LLM judge; if judges favor their own outputs, benchmarks systematically tilt toward that model family.

Related event: Stanford Study Finds All 8 LLM Judges Favor Their Own Medical Answers(3 posts)→

Original post →

More from Models

Models channel →