Stanford's 13,760-judgment study exposes LLM judge self-preference in medical evals
rohanpaul_ai · x · 2026-11-18
The Stanford paper "The Judge Is Not Impartial: Self-Preference in Medical LLM Evaluation" (arXiv:2611.18616) systematically quantifies self-preference when the same model both generates and judges medical content.
Setup
- 8 models (two each from OpenAI, Anthropic, Google, Qwen) answered 620 real point-of-care questions (Real-POCQi) and generated/judged 200 standardized patient scenarios (MedSP1000) up to 8 turns
- 13,760 LLM judgments analyzed; three measures: Answer Rank Bias, Answer Score Bias, Own-Answer Win Rate
Findings
- Pooled Answer Rank Bias of +1.22 positions; all 8 judges favored their own answers
- Bias persisted with and without a grading rubric
- Revealing generator identities weakened it by only 0.094 positions
- MedSP1000 bias ranged +0.53 to +0.62, independent of conversation length (p=0.17)
Recommendation: use judge panels spanning model families and multiple evaluation measures to break the circularity in clinical AI evaluation.
Related event: Stanford Study Finds LLM Judges Favor Their Own Medical Answers(2 posts)→
More from Models
- AI daily: GPT-6.2 launch, NVIDIA revenue doubles, AI singer tops Korean charts — xiaohu · 2026-11-18
- Early GPT-6.2 Astra & Sol hands-on: delegating whole workflows 'feels like a step change' — reach_vb · 2026-11-18
- After September's GPT quality slump, users doubt 6.2 restores reliability — Beautiful_Cheetah789 · 2026-11-18
- Indie dev: skip GPT-6.2 hype, what I want is a better Antigravity harness — yihui_indie · 2026-11-18
- New Gemini Astra shows cool behavior in novelty search harness — generativist · 2026-11-18
- Stanford finds all 8 LLM judges favor their own medical answers, skewing benchmarks — rohanpaul_ai · 2026-11-18