Adding more eval samples won't fix prompt-choice variance, paper shows
srchvrs · x · 2026-11-18
A variance model for LLM evaluation decomposes outcome variance into task variability, prompt/judge configuration variability, and residual noise. The key observation: as long as you evaluate with a single prompt configuration, increasing the number of test examples never reduces the uncertainty from prompt choice. Robust conclusions across reasonable prompts, judges, and harnesses require accounting for configuration-level variability too.
Related event: Researcher: LLM Eval Statistics Must Account for Prompt and Judge Variance(4 posts)→
More from Research
- UChicago study: Pangram AI-text detector beats GPTZero, Originality and RoBERTa with <5% error rates — _akpiper · 2026-11-18
- ITERLOGIC uses iterative natural logic verification to cut hallucinations in multi-hop QA — chrmanning · 2026-11-18
- Ofir Press on Kaggle podcast: benchmark design, LM-as-a-Judge, and whether coding is solved — OfirPress · 2026-11-18
- Are looped models necessarily less interpretable? A researcher pushes back on the common claim — torchcompiled · 2026-11-18
- Amazon's CompoundingKernel: 1000+ agents write GPU kernels from scratch, tops 3 SOL-ExecBench boards — dair_ai · 2026-11-18
- Planted "hand" direction probe shows Qwen3's latent body map is tuning-gated — repligate · 2026-11-18