When LLM judges agree, should we believe them? Amazon Science says not necessarily
Betelbuddy · hn · 2026-09-15
Amazon Science challenges a common assumption in LLM-as-a-judge setups: agreement among multiple LLM judges doesn't mean correctness. LLM evaluators built on similar models and data can share systematic biases, so consensus may reflect shared bias rather than truth. The piece analyzes why and suggests ways to mitigate this when using LLMs for evaluation.
More from Research
- ELLIS x UniReps talk: why distinct learning processes yield similar representations — ClementineDomi6 · 2026-10-30
- Physicist calls OpenAI's Navier-Stokes solution physically meaningless despite loophole win — GaryMarcus · 2026-10-30
- Anthropic paper: 0.1% pretraining poison plants RCE backdoor in 4B CLI agent — omarsar0 · 2026-10-30
- CopeNLU presents 7 papers at EMNLP 2026 on bias, alignment, explainability — IAugenstein · 2026-10-30
- Two-stage 3D ResNet pipeline detects neonatal brain pathologies from MRI with 0.919 AUC — Dr_Alex_Crimi · 2026-10-30
- 40 years of putting new neurons into the CNS: why so slow, and can AI accelerate it? — patrickmineault · 2026-10-30