Prompt Variance Caps LLM Eval Accuracy No Matter How Many Samples You Add

srchvrs · x · 2026-11-18

srchvrs lays out a key blind spot in LLM evaluation methodology: conventional statistical tests are fine for a fixed model, prompt, judge, and harness, but conclusions meant to hold across reasonable prompts, judges, and harnesses require accounting for variability along those dimensions — which existing tests don't capture.

An accompanying variance-decomposition model shows the mean's variance splits into task variance/N, prompt-judge configuration variance/M, and residual/NM, with the crucial observation that increasing N does not reduce the prompt-variance term.

Related event: Researcher: LLM Eval Statistics Must Account for Prompt and Judge Variance(4 posts)→

Original post →

More from Research

Research channel →