Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Sho Kawano, Zehang Richard Li, Paul A. Parker
stat.ML, cs.AI, cs.LG, stat.AP, stat.ME
2026-09-18
PP-TS, a Bayesian Fay-Herriot smoother on GREG domain estimates, cuts phi-4's 34-task RMSE from 0.082 to 0.059 at a 10% label budget, with 93-95% interval coverage.
Disaggregated scores are now the default: HELM-style suites split by task type, live agents split by conversation type or user segment. Gold labels are expensive. Liang et al. put a single LLM run of HELM at $9,337 in API credits; open-ended work adds human grading. Under a tight budget each domain gets few labels.
Direct estimators use only a domain's own sample. Horvitz-Thompson (HT, inverse-probability weighted mean), prediction-powered inference (PPI, a full-population prediction plus a sampled residual correction), and generalized regression (GREG, PPI++ with a data-estimated correction slope) all sit in this class. Variance scales as \(1/ni\). Small area estimation exists for exactly this: borrow strength across domains. What was missing is a workflow that does point estimates, intervals, and a choice between direct and smoothed estimators from one sample.
Treat the evaluation set as a finite population. The target \(\thetai\) is the mean over every unit in domain \(i\), not a superpopulation parameter. Sampling has to be a probability sample with known inclusion probabilities. Section 2 shows the alternative on PRISM: oversampling rejected replies and then estimating as if the draw were random leaves a bias that does not vanish with budget, and 95% interval coverage falls well below nominal. PPI does not repair this, because the residuals inherit the same selection bias.
Given a probability sample, the workflow has two stages.
Estimation. Compute GREG in each domain: the HT mean plus a correction that uses an auxiliary observed on the whole population (historical model correctness, item difficulty, an LLM-judge score). The slope \(\hat\lambda\) comes from one weighted regression pooled across domains. Weak auxiliaries drive \(\hat\lambda\) down to about one third, so GREG does not degrade relative to HT; PPI has no such knob and its RMSE rises above HT when the auxiliary is weak. Feed the GREG point estimates and their sampling variances into a Bayesian Fay-Herriot model, which shrinks each estimate toward a domain-level regression in proportion to noise. That is prediction-powered smoothing (PP-S). If domains nest in a taxonomy (task type inside a benchmark, or LLM crossed with conversation type), random effects add by level: PP-TS. As a domain's sample grows, the smoother's weight on the direct estimate goes to 1, so design consistency is kept.
Validation. Direct estimators are unbiased; smoothed ones trade bias for variance, so sampling variance alone cannot rank them. The new design-based cross-validation (DB-CV) splits the sample into \(K\) folds within domains, scores each candidate against the held-out HT estimate, then subtracts fold-level noise and the full-sample HT error to get an approximately unbiased MSE. Covariances have closed forms for HT, GREG, and the smoothers. One sample both selects a candidate and reports that candidate's error.
Both datasets have every outcome observed, so the oracle is known. Labels are drawn by stratified simple random sampling, 10% budget, 100 replications.
Open LLM Leaderboard: 9,324 questions, 34 task types nested in five benchmarks, estimand is microsoft/phi-4 accuracy. Auxiliaries come from historical records that exclude phi-4. Unit-level correlations: 0.31 for Qwen2.5-Math-7B-Instruct, 0.33 for Qwen2.5-1.5B-Instruct, 0.61 for historical difficulty.
| Estimator | Specialist RMSE | Generalist RMSE | Difficulty RMSE |
| HT | 0.082 | 0.082 | 0.082 |
| PPI | 0.106 | 0.105 | 0.074 |
| GREG | 0.080 | 0.082 | 0.074 |
| PP-S | 0.075 | 0.071 | 0.061 |
| PP-TS | 0.072 | 0.070 | 0.059 |
PP-TS has the lowest RMSE and interval score under every auxiliary (IS 0.295 vs HT 0.418 on difficulty). Coverage sits between 0.93 and 0.95. Shrinkage alone, Fay-Herriot with no auxiliary, barely beats HT. Most of the gain is the domain-level covariate; taxonomy is next. The two weak auxiliaries have almost the same unit-level correlation, yet the generalist's domain mean cuts RMSE far more than the specialist's. The specialist is accurate on MATH and weak on IFEval; taxonomy absorbs that unevenness.
PRISM is treated as live traffic: 68,371 ratings on a 1-100 scale, 63 domains (21 LLMs times 3 conversation types). The judge is gpt-5-nano, unit correlation 0.40, domain-mean correlation 0.93, full-population scoring about six dollars. At 10% (6,837 labels) PP-S with judge plus content covariates has oracle RMSE 1.527 against HT 2.505. In the sparsest quarter of domains HT intervals average 15 points and PP-S about 7, and those domains also have the lowest satisfaction.
At the same budget DB-CV's chosen estimator has oracle RMSE 1.543, Kendall \(\tau\) 0.89, and reported error 1.07 times oracle. Naive CV picks almost as well (1.528) but reports error at 4.00 times. 50/50 and 80/20 splits report 2.59 times and 3.54 times. The 20% budget repeats the pattern.
Teams doing disaggregated eval usually hold one labeled sample and a pile of cheap predictions. This paper wires PPI/GREG to small-area models and adds a score that selects and reports error from that single sample. Code and processed tables are public.
The operating constraint is sharp: a probability sample is required. Benchmark owners control the draw, so this is feasible. Traffic selected by complaints, flags, or reviewer attention will bias the estimate, and the labels cannot audit that bias. Smoothing by itself is a small term; how the auxiliary is used at domain level decides the gain. Two equally accurate LLM judges can differ as covariates, which is why the validation step is not optional.
This is survey statistics brought into LLM evaluation, a working pipeline, not a new eval protocol.
The authors flag several gaps. Zero-shot domains with no labels but available covariates have no reliability criterion yet. Open-ended grades are judgments; rater disagreement is not folded into the intervals. How to combine large nonprobability samples (flagged conversations) with a probability sample is left as a pointer.
Both populations are held fixed; randomness is only which units are sampled. Live traffic drifts, and coverage under shift is untested. The Fay-Herriot sampling model is a CLT approximation, weakest in the smallest domains, which are the ones that borrow most. In Table 4 naive CV's chosen RMSE (1.528) is slightly better than DB-CV's (1.543); DB-CV's claim is calibrated error reporting, not strictly better selection.