Flowchart says LLM judges need at least 15 labels and IRR around 0.40

IanArawjo · x · 2026-07-29

A decision flowchart proposes when LLM judges are worth using in mixed human-AI evaluation studies, and when researchers should fall back to classical human-only statistics.

Key points:

First ask whether AI judges are worth the trouble at all: large datasets and a separate tuning set make them more viable.
Randomly sample items for human labels, keeping the judge aligned on a tuning set.
The flowchart recommends a minimum labeled sample size of Nlabl ≥ 15 and an inter-rater reliability threshold of about IRR ≥ 0.40 before trusting judge + human data.
If the data are too small, or if the metric is easy to code automatically, the advice is to avoid LLM judges and revert to classical statistics on human-only or coded metrics.
Even with high IRR, the figure stresses that this is not a license to skip PPI correction; researchers should run the appropriate PPI-corrected test and report judge model, prompt/setup, alignment metric, and results.

Original post →

Flowchart says LLM judges need at least 15 labels and IRR around 0.40

More from Research