LLM Judges Unreliable Without Human Validation
IanArawjo · x · 2026-07-19
The author points out that if you only use LLMs to score missing human data, the results are just as bad as using a biased LLM judge directly, significantly inflating the false positive rate.
They ran simulations across a wide range of data distributions and found that uncorrected LLM judgments easily produce "results that look significant but are actually false." Conversely, averaging 4 statistical tests corrected with PPI maintains good false-positive control even when relying on only a small number of human labels. The conclusion is straightforward: LLM judge results without human validation are untrustworthy.
Related event: Study Warns: Unchecked LLM Judges Yield High False Positives(7 posts)→
More from Research
- New theory paper explains what Anthropic's Jacobian lens actually detects in LLMs — wgilpin0 · 2026-10-27
- Interpretability researcher joining Goodfire AI post-PhD, calls for speeding up the field — mathildepapillo · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
- Latent-Foresight: world models that learn to predict their own future — abursuc · 2026-10-27
- UK AISI papers: no good safeguards yet against misuse of biological AI tools — davidmanheim · 2026-10-27