Researcher flags prompt variance as blind spot in LLM evaluation statistics

Researcher srchvrs argues that while classic statistical tests over independently sampled tasks remain valid for fixed model, prompt, judge and harness, conclusions must account for variation across prompts, judges and harnesses — and simply adding more tasks won't reduce prompt-variance error.

2026-11-18 ~ 2026-11-18 · 2 related posts