High Inter-Rater Consistency in LLM Evaluation May Be Misleading, Researcher Warns

Ian Arawjo argues that high inter-rater consistency in LLM-as-Judge evaluation does not guarantee quality, calling the common practice misleading. He adds that widely recommended metrics such as Spearman's rho, Krippendorff's alpha and Kappa suffer from the same flaw.

2026-08-25 ~ 2026-08-25 · 2 related posts

• High IRR LLM judges risk more false positives, study warns — IanArawjo · 2026-08-25
• Spearman and Kappa metrics misleading for LLM judge QC — IanArawjo · 2026-08-25