Researcher flags prompt variance as blind spot in LLM evaluation statistics
Researcher srchvrs argues that while classic statistical tests over independently sampled tasks remain valid for fixed model, prompt, judge and harness, conclusions must account for variation across prompts, judges and harnesses — and simply adding more tasks won't reduce prompt-variance error.
2026-11-18 ~ 2026-11-18 · 2 related posts
- Eval methodology: statistical tests are valid, but conclusions must span prompt, judge, and harness variance — srchvrs · 2026-11-18
- Prompt Variance Caps LLM Eval Accuracy No Matter How Many Samples You Add — srchvrs · 2026-11-18