Eliminating LLM Evaluation False Positives with PPI-Corrected Tests

Researcher Ian Arawjo calls on the industry not to blindly trust uncalibrated LLM evaluation results, noting that this easily leads to false positives and false conclusions. His team is introducing PPI (Prediction-Powered Inference) corrected statistical tests via evalstats, which can effectively correct LLM judge evaluation biases by simply inputting a small number of human labels.

Confirmed

The PPI correction method has been validated on real LLM judge data with human labels. The current analysis utilized three datasets: WMT translation quality, Chat (related data), and others. When judge bias exists, uncorrected standard tests may produce false positives (shown by grey points in the chart), whereas adding partial human labels allows the correction method to effectively reduce this bias (shown by colored points).

Unconfirmed

While the PPI corrected test performs well in simulated environments, non-normal statistical methods may encounter issues when facing real data (especially poorly aligned and low-quality evaluation scores). The author is currently investigating this. Furthermore, the Wilcoxon signed-ranks test appears "conservative" in some test charts, but this is not an inherent property of the method; rather, it is an artifact caused by the judge score distribution of the specific test data.

Why it matters

As LLM evaluations increasingly rely on models as judges, the issue of false positives severely threatens the reliability of evaluation conclusions. Introducing a small number of human labels for PPI correction provides the industry with a statistical bias correction solution that balances cost and accuracy, though the method's robustness on extreme real-world data requires continuous validation.

2026-07-24 ~ 2026-07-24 · 5 related posts

Primary sources

evalstats: Eliminating False Positives in LLM Judging with PPI-Corrected Tests — IanArawjo · 2026-07-24 10:13 (UTC+08:00)
PPI-corrected tests aim to fix false positives in LLM judge evaluations — IanArawjo · 2026-07-24 10:15 (UTC+08:00)
PPI-corrected tests show how human labels reduce LLM judge bias — IanArawjo · 2026-07-24 10:13 (UTC+08:00)

Challenges of Non-Parametric Statistical Tests on Real-World Data — IanArawjo · 2026-07-24
[source] PPI-corrected tests show how human labels reduce LLM judge bias — IanArawjo · 2026-07-24
[source] evalstats: Eliminating False Positives in LLM Judging with PPI-Corrected Tests — IanArawjo · 2026-07-24
Wilcoxon signed-ranks looks conservative only because of this test dataset — IanArawjo · 2026-07-24

1 near-duplicate retellings: IanArawjo