Eval methodology: statistical tests are valid, but conclusions must span prompt, judge, and harness variance
srchvrs · x · 2026-11-18
IR/eval researcher srchvrs offers a two-point methodological clarification on LLM benchmarking: (1) conventional statistical tests across independently sampled tasks are entirely appropriate when model, prompt, judge, and harness are fixed; (2) conclusions meant to hold across reasonable prompts, judges, and harnesses must account for variability along those dimensions—which existing tests do not capture. A technical critique of how benchmark conclusions get over-generalized.
Related event: Researcher flags prompt variance as blind spot in LLM evaluation statistics(2 posts)→
More from Research
- RL lecturer shares lecture slides: supervised learning approximates, RL discovers functions — KyleMorgenstein · 2026-11-18
- Supervised vs self-supervised is a false dichotomy: RL is where it ends up — KyleMorgenstein · 2026-11-18
- "Supervised vs self-supervised" is a false dichotomy, argues UT RL lecturer — KyleMorgenstein · 2026-11-18
- ELLIS Sites NeurIPS Fest 2026 returns across Europe with 18 accepted NeurIPS papers — DocXavi · 2026-11-18
- Decompiled Instagram APK reveals on-device models and probing for 23 installed AI apps — ChenhaoTan · 2026-11-18
- Running a 180B MoE Model on a 16GB MacBook: Experts Offloaded to SSD, 14× Faster Than llama.cpp — XiongChenyan · 2026-11-18