Eval methodology: statistical tests are valid, but conclusions must span prompt, judge, and harness variance

srchvrs · x · 2026-11-18

IR/eval researcher srchvrs offers a two-point methodological clarification on LLM benchmarking: (1) conventional statistical tests across independently sampled tasks are entirely appropriate when model, prompt, judge, and harness are fixed; (2) conclusions meant to hold across reasonable prompts, judges, and harnesses must account for variability along those dimensions—which existing tests do not capture. A technical critique of how benchmark conclusions get over-generalized.

Related event: Researcher flags prompt variance as blind spot in LLM evaluation statistics(2 posts)→

Original post →

More from Research

Research channel →