Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation
Mingqi Gao, Anthony Sicilia, Weiyan Shi
EMNLP 2026 (Main)
cs.CL, stat.ML
2026-08-27
On six WMT sets, a few human labels debias metric scores. GEMBA-only coverage is ~30%; PPI restores ~95%, narrower than human-only. PPSR separates metrics better than SPA.
Comparing translation systems means estimating the true gap in human scores. Human ratings are the reference, and the samples are often too small to give a test much power. Automatic metrics and LLM judges are cheap and can score every segment, but they are biased, so a metric-only decision can pick the wrong winner. The two evaluations are usually reported separately.
Prediction-powered inference (PPI) cuts variance with a large batch of cheap predictions and keeps the estimate unbiased with a small batch of real labels. On system comparison, the metric's job is to save human labels. Three WMT habits do not match off-the-shelf PPI. The default test is a Z-test, while shared tasks have long preferred permutation tests, the bootstrap, and Wilcoxon. The default design is paired, while some years the two systems are not scored on the same sources. Meta-metrics rank metrics by agreement with humans. With the tuning weight held fixed, a PPI estimate is unbiased anyway, so the quantity to rank is how many labels the metric saves.
For systems scored on the same sources, the estimator averages metric gaps on unlabeled segments and adds a correction on labeled ones: human gap minus λ times metric gap. Any fixed λ leaves the estimate unbiased. λ set to 0 is ordinary human-only evaluation. At the covariance-optimal λ, variance is no higher than human-only, and nothing is saved if the human and metric gaps are uncorrelated. A badly biased metric does not break unbiasedness, and the two scores need not share a scale. Optimal λ moves with the covariance and gets smaller as labeled size grows relative to unlabeled size. L is the number of segments with human scores, U the number with only metric scores. λ is fit on the labeled sample and is not clipped to the unit interval. Intervals use a paired Z-test from the central limit theorem.
That Z-test needs a normal approximation and an accurate variance. Human scores are discrete, and a few dozen labels is common, so both fail often. The paired permutation test is extended to the PPI statistic: each segment gets a random sign, the human gap and the metric gap flip together, and λ is re-estimated on every shuffle. Metric gaps are centered first. The test asks that the pair of gaps be symmetric about their means, and it does not use a normal approximation. With centering, Type I error stays near the nominal 0.05 even when the metric is biased.
When sources do not match, each system gets its own PPI estimate and the estimates are subtracted. The difference is still unbiased. Which design has the smaller variance depends on how the human scores covary, and on whether the metric tracks score gaps or the individual scores.
PPSR turns that saving into a meta-metric. When U is much larger than L, the fraction of variance removed is about the squared Pearson correlation of the two gaps on that system pair. The same fraction is the share of human labels that can be dropped while power stays about the same. Averaged over system pairs it lies between 0 and 1, and it uses only labeled data. A PPSR of 0.4 means about 40% fewer labeled segments. Because the correlation is squared, a metric that moves against human scores can still save labels.
The six sets, in order, are WMT22 English-German, English-Russian, and Chinese-English; WMT23 English-Chinese and Japanese-English; and WMT24 Czech to Ukrainian. The first three use MQM, with 1315, 1315, and 1875 inputs, 14, 15, and 14 systems, and 31, 30, and 31 metrics. The next two use DA-SQM, with 1098 and 1120 inputs, 15 and 17 systems, and 34 and 35 metrics. Czech to Ukrainian uses ESA, with 1955 inputs, 11 systems, and 25 metrics. Labeled size runs from 20 to 200, unlabeled size is fixed at 800, and each cell is drawn 1000 times. The human gap on the full set is the population effect. The working metric is that year's MetricX.
On Czech to Ukrainian with 80 labels, MetricX alone has lower power than the human-only test for some true effect sizes. The metric test sees all 880 segments; the human test sees 80. If MetricX were unbiased, that ordering should not happen. PPI power tracks the human curve and stays above it. The permutation version of PPI also beats human-only permutation, with less power than the Z-test. GEMBA, an LLM judge, yields very narrow automatic 95% intervals with empirical coverage around 30%. The same scores inside PPI cover close to 95%, and the intervals stay narrower than human-only. MetricX and BLEU also narrow the interval, MetricX by more, with coverage near 95% for both.
At 20 labels the PPI Z-test is anti-conservative. Under a heavy-tailed null with correlation 0.7 and 3 degrees of freedom, its Type I error is 0.123 against a nominal 0.05. Under a normal null it is still 0.104. Plug-in variance, not only the normal approximation, is what inflates it. The permutation test stays near 0.05. The Z-test is more powerful and a poor default at this sample size. The human-only Z-test reaches at most about 0.059.
Dropping the pairing inflates variance. In human-only evaluation the average increases, in the dataset order above, are 1.28, 1.81, 1.28, 0.16, 0.18, and 0.56, positive for 98% to 100% of system pairs. Under PPI they fall to 1.11, 1.67, 1.02, 0.11, 0.12, and 0.33, and 92% to 100% of cells are still positive. Pair the systems when they share sources. The metric usually correlates more with each system's human score (mean absolute correlation 0.27 to 0.43) than with the gap (0.17 to 0.25), so the paired advantage shrinks once the metric is included.
PPSR finds 412, 373, 422, 507, 500, and 247 significant metric-versus-metric differences. Pearson finds 355, 354, 402, 493, 389, and 234. SPA finds 306, 314, 358, 440, 338, and 225. On distinct values, PPSR, Pearson, and SPA all separate every metric; rank correlations still tie often. Redraws of 100 to 1000 inputs, 1000 times each, leave PPSR's ranking closest to the full-data ranking. Correlation with the number of labels actually saved to hit 80% power is 0.987, 0.957, 0.966, 0.859, 0.974, and 0.989 on the six sets. The PPSR scatter sits mostly under 0.2.
Hold λ fixed and the system gap stays unbiased, so metric bias does not pass straight into the comparison. GEMBA's automatic intervals cover the truth only about 30% of the time. A narrow interval can miss the truth. PPI brings coverage back near the nominal 95% and still undercuts the human-only width.
System-level correlation averages a dozen systems first, so ties pile up. On significant pairwise tests SPA trails PPSR. PPSR is denominated in labels saved. Sharper rankings do not make it a substitute for agreement metrics, which answer a different question. On these sets the typical metric saves well under a fifth of the human labels.
Four limits are stated in the paper. Annotator disagreement is left out of the human variance. The same labeled sample is used to fit λ and to estimate the interval, with λ treated as known, which makes the variance look too small. That is not corrected. System scores are assumed to be means over inputs, so other aggregations do not fit. Everything is tested on machine translation. Other non-verifiable tasks are not.
Power and coverage come from resampling annotations that already exist, not from a new labeling round. Unlabeled size is locked at 800, and the squared-correlation reading of PPSR wants far more unlabeled segments than labeled ones. Joint symmetry is assumed by the permutation test and not checked on the WMT gaps. Type I error is measured on synthetic draws. At 20 labels the parametric test exceeds its nominal rate by more than a factor of two.