High Consistency Across LLM Judge Ensembles Can Amplify Bias, Study Warns
IanArawjo reported that an ensemble of 7 LLM judges achieved high inter-rater agreement yet produced entirely spurious results, because correlated model biases amplify systematic error when noise is low.
2026-09-07 ~ 2026-09-07 · 2 related posts
- LLM judge ensemble hit high IRR yet produced a bogus super-significant result — IanArawjo · 2026-09-07
- Why LLM judge ensembles amplify bias: correlated models boost systemic errors — IanArawjo · 2026-09-07