Prompt Variance Caps LLM Eval Accuracy No Matter How Many Samples You Add
srchvrs · x · 2026-11-18
srchvrs lays out a key blind spot in LLM evaluation methodology: conventional statistical tests are fine for a fixed model, prompt, judge, and harness, but conclusions meant to hold across reasonable prompts, judges, and harnesses require accounting for variability along those dimensions — which existing tests don't capture.
An accompanying variance-decomposition model shows the mean's variance splits into task variance/N, prompt-judge configuration variance/M, and residual/NM, with the crucial observation that increasing N does not reduce the prompt-variance term.
Related event: Researcher: LLM Eval Statistics Must Account for Prompt and Judge Variance(4 posts)→
More from Research
- Planted "hand" direction probe shows Qwen3's latent body map is tuning-gated — repligate · 2026-11-18
- Ropedia Xperience 2: 1,500+ real scenes for scaling physical AI training data — _akhaliq · 2026-11-18
- Manning prints 'CUDA for LLMs,' a 408-page guide to fast transformer kernels, 50% off till Dec 4 — eigenron · 2026-11-18
- CMU posts free 'Machine Learning with Large Datasets' lectures on YouTube weekly — phcompeau · 2026-11-18
- LAION-Audio-XL Tops Hugging Face Trending Audio Dataset Charts — wightmanr · 2026-11-18
- 516-config meta-study: multi-agent LLMs underperform matched baselines by 4 points on average — Seyed Mahmoud Sajjadi Mohammadabadi · 2026-11-18