Stanford finds all 8 LLM judges favor their own medical answers, skewing benchmarks
rohanpaul_ai · x · 2026-11-18
A new Stanford paper shows systematic self-preference in LLM-as-judge medical evaluation: 8 models from OpenAI, Anthropic, Google and Qwen answered 620 real clinician questions, then anonymously ranked all answers. Every judge ranked its own model's answer higher, by +1.22 rank positions on average and nearly 3 for GPT-5.6 Sol.
- The bias persisted with or without a grading rubric
- Revealing generator identities weakened it by only 0.094 positions
- Medical benchmarks using LLM judges are therefore tilted toward self-evaluating models
The authors recommend using judge panels spanning multiple model families and reporting results per judge rather than pooling scores.
Related event: Stanford Study Finds LLM Judges Favor Their Own Medical Answers(2 posts)→
More from Models
- AI daily: GPT-6.2 launch, NVIDIA revenue doubles, AI singer tops Korean charts — xiaohu · 2026-11-18
- Early GPT-6.2 Astra & Sol hands-on: delegating whole workflows 'feels like a step change' — reach_vb · 2026-11-18
- After September's GPT quality slump, users doubt 6.2 restores reliability — Beautiful_Cheetah789 · 2026-11-18
- Indie dev: skip GPT-6.2 hype, what I want is a better Antigravity harness — yihui_indie · 2026-11-18
- New Gemini Astra shows cool behavior in novelty search harness — generativist · 2026-11-18
- Dev pushes back: model intelligence is solving hard problems, not flashy 3D demos — GabGarrett · 2026-11-18