Adding more eval samples won't fix prompt-choice variance, paper shows

srchvrs · x · 2026-11-18

A variance model for LLM evaluation decomposes outcome variance into task variability, prompt/judge configuration variability, and residual noise. The key observation: as long as you evaluate with a single prompt configuration, increasing the number of test examples never reduces the uncertainty from prompt choice. Robust conclusions across reasonable prompts, judges, and harnesses require accounting for configuration-level variability too.

Related event: Researcher: LLM Eval Statistics Must Account for Prompt and Judge Variance(4 posts)→

Original post →

More from Research

Research channel →