Stanford study: all 8 LLM judges rank their own medical answers higher, by 1.2 spots on average
TheTuringPost · x · 2026-11-19
A Stanford paper tested 8 models from OpenAI, Anthropic, Google and Qwen: each answered 620 real clinician questions, then all 8 anonymously ranked every answer.
- Every judge ranked its own answer higher than the other 7 judges did — by 1.22 rank positions on average, nearly 3 for GPT-5.6 Sol
- The bias held without a rubric, and revealing which model wrote which answer barely changed it
The authors note many weekly leaderboards rely on a single LLM judge; if judges favor their own outputs, benchmarks systematically tilt toward that model family.
Related event: Stanford Study Finds All 8 LLM Judges Favor Their Own Medical Answers(3 posts)→
More from Models
- Astra's proof sidesteps three deep Mathlib results by inventing a new 'capacity' quantity — IgorCarron · 2026-11-19
- Ex-Claude superfan: getting the best from Astra/Sol requires relearning how you prompt — mariofilhoml · 2026-11-19
- Observer: Google and Anthropic are winning the public AI race, OpenAI's lead now in doubt — imjustnewatai · 2026-11-19
- Three Google flagship models in eight days fuel speculation about hidden RSI work — haider1 · 2026-11-19
- Mystery Model 'Gestalt' Tops New Frontier Design Benchmark by 214 Points — soleio · 2026-11-19
- mrfakename releases abliterated GLM-5.3-Flash and Qwen3.8-Flash-Next — mrfakename0 · 2026-11-19