Why a high kappa score still does not make an LLM judge trustworthy
IanArawjo · x · 2026-07-21
A follow-up explanation of the same metric issue: agreement statistics are more sensitive to **random disagreements** than to a **consistent offset**. If an LLM is biased by a nearly constant amount — for example, always scoring one point above or below human raters — inter-rater alignment metrics may still look fine. That means a high kappa score does not necessarily mean the judge is truly reliable. ### Takeaway - Random noise hurts agreement scores more than stable bias. - A judge can look “aligned” while still being systematically off in practice.
Related event: Study Warns of High False Positive Rates in Unchecked LLM Judges(7 posts)→
More from Research
- Gemma 4 12B visualization shows what the model predicts from video patches — arjunrajlab · 2026-07-21
- Baseten study finds new facts in LLM weights are fragile unless trained from many restatements — alex_verem · 2026-07-21
- Kimi K3 and Fable 5 now look much closer than the old open-vs-closed gap — FinanceYF5 · 2026-07-21
- uv-scripts/ocr returns to the top of Hugging Face datasets with a JSON model picker — vanstriendaniel · 2026-07-21
- DeepSearch-World trains web agents with 420K verifiable QA tasks — HKUST · 2026-07-21
- GigaAM Multilingual targets low-resource Central Asian ASR with 2M hours of audio — ai-sage · 2026-07-21