LLM judges are strongly biased toward position, researcher warns to randomize everything
antoine_chaffin · x · 2026-10-31
antoinechaffin highlights hammermt's finding that LLM-as-a-judge setups carry a strong position bias: judge models often just pick whichever answer comes first.
His methodological takeaway:
- When evaluating decider/judge models, randomize every exploitable signal (order, position, format)
- Otherwise evaluations contain signals that models can exploit but that do not hold on real traffic, leading to misleading quality assessments
A useful warning for anyone building LLM-as-a-judge eval pipelines: always shuffle candidate order to avoid systematic bias.
More from coding & agent
- LambdaTest's TestMu AI Mapped: From Test Cloud to Agent Verification — Designer_Hair_417 · 2026-10-31
- Open-source ultra-elements: 39 tunable spell VFX in pure three.js, LLM-extensible — majidmanzarpour · 2026-10-31
- How Boardy made CI 20x faster: PR waits drop from 40 minutes to 2 — inductionheads · 2026-10-31
- Product Engineering vs engineer-centric AI coding: ship all the way to paying users — PawelHuryn · 2026-10-31
- One start-here skill navigates you through the 43-skill agent workflow — PawelHuryn · 2026-10-31
- hbench visualizes agent trajectories — and caught DeepSeek reading benchmark answers — Psychological_Cry920 · 2026-10-31