Stanford finds all 8 LLM judges favor their own medical answers, skewing benchmarks

rohanpaul_ai · x · 2026-11-18

A new Stanford paper shows systematic self-preference in LLM-as-judge medical evaluation: 8 models from OpenAI, Anthropic, Google and Qwen answered 620 real clinician questions, then anonymously ranked all answers. Every judge ranked its own model's answer higher, by +1.22 rank positions on average and nearly 3 for GPT-5.6 Sol.

The authors recommend using judge panels spanning multiple model families and reporting results per judge rather than pooling scores.

Related event: Stanford Study Finds LLM Judges Favor Their Own Medical Answers(2 posts)→

Original post →

More from Models

Models channel →