LLM eval conclusions don't generalize unless you model prompt/judge/harness variability

srchvrs · x · 2026-11-18

A key methodological distinction for LLM evaluations: with a fixed model, prompt, judge, and harness, conventional statistical tests across independently sampled tasks are perfectly appropriate. But if you want conclusions to hold across reasonable prompts, judges, and harnesses, you must also account for variability across these configurations — single-setup significance doesn't equal a general finding.

Related event: Researcher: LLM Eval Statistics Must Account for Prompt and Judge Variance(4 posts)→

Original post →

More from Research

Research channel →