LLM judges favor their own company's models on close debate questions
Trainer_28 · reddit · 2026-10-13
A developer running two-model debates with a third model as judge found consistent self-preference bias: on close questions, judges pick the model from their own company. Same judge + same transcript = same verdict every time, and it persists even when the last turn is removed. Models can't see names—only style.
Across 230 debates, GPT judges gave GPT 78% of wins against Claude, but Claude judges actually gave GPT 56%—and 83% in GPT vs Gemini matchups—suggesting stronger models usually win regardless, with bias only surfacing on tight calls.
Mitigations tried: reframing the judge prompt from "who argued better" to "what is the right answer" (partially fixed), a third-party Gemini judge (unstable), and finally a Claude+GPT panel with Gemini as tiebreaker. Cites self-preference bias papers arXiv:2410.21819 and 2404.13076.
More from coding & agent
- Dev spins up a horde of Claudes to hit a one-week deadline — "it worked out" — rickasaurus · 2026-10-13
- Debug Multi-Agent Workflows: Permissions, Handoffs and Channels Beat Model Blame — Al_Grigor · 2026-10-13
- iOS Dev Mocks the AI-Agent Era: HomeView, agents.md, Skills and App Store Rule 2.3bc — Dimillian · 2026-10-13
- Who decides when AI evaluator and human reviewer disagree on an agent call? — AssociationNew7925 · 2026-10-13
- Agentic coding solves the easiest part of engineering and breaks the hardest — cgouguen · 2026-10-13
- Dev builds real-time dictation pipeline: AssemblyAI Sync API plus LLM Gateway rewrite — tech__unicorn · 2026-10-13