Researcher Warns LLM-as-Judge Shows Position Bias; Randomize Everything in Evaluations

Mistral researcher Antoine Chaffin warns that LLM-as-judge suffers strong position bias—often picking the first answer rather than the better one—and advises randomizing everything when evaluating decider models to avoid exploitable, fake signals.

2026-10-31 ~ 2026-10-31 · 3 related posts

• LLM judges are strongly biased toward position, researcher warns to randomize everything — antoine_chaffin · 2026-10-31
• Eval researcher: randomize everything for decider models, exploit signals don't hold in real traffic — antoine_chaffin · 2026-10-31
• Mistral researcher: randomize everything when evaluating decision models — antoine_chaffin · 2026-10-31