LLM judge ensemble hit high IRR yet produced a bogus super-significant result
IanArawjo · x · 2026-09-07
IanArawjo recounts an empirical pitfall: ensembling 7 LLM judges to predict human scores on a dataset yielded a high inter-rater reliability that looked great — but produced a totally bogus "super-significant" result. Seeing that highly aligned LLM judges raise false positives in the abstract is very different from hitting it in practice.
Related event: High Consistency Across LLM Judge Ensembles Can Amplify Bias, Study Warns(2 posts)→
More from Research
- Spotting reward hacking in benchmarks is far harder than you think, says AI2 researcher — soldni · 2026-10-26
- RCT: ChatGPT temporarily boosts consultants' data science skills, gains vanish without it — EricTopol · 2026-10-26
- FocusVLA routes robot camera views by predicted interaction intent across task stages — iscas-cloud · 2026-10-26
- Transformers' biggest audio overhaul: unified AudioProcessor architecture — LysandreJik · 2026-10-26
- Free 195-page primer: The Little Book of Generative AI Foundations — caglar_ee · 2026-10-26
- SPADER from Zhejiang and Peking University Tops All Four Multi-Answer QA Benchmarks, Accepted at EMNLP — jiqizhixin · 2026-10-26