Preregistered Analysis and Eligible Model Sets Prevent Eval p-Hacking
krisgligoric · x · 2026-07-06
A new protocol has been proposed to prevent p-hacking in LLM evaluations: preregister an analysis plan alongside a set of "eligible models," then run confirmation experiments on the first eligible LLM released after preregistration. Because the model did not exist at the time of commitment, it cannot be specifically gamed, mechanically eliminating the manipulation of evaluations.
Related event: Study Proposes Preregistration Protocol to Curb LLM Evaluation P-Hacking(7 posts)→
More from Research
- OpenAI internal model claims to refute Exponential Time Hypothesis; independent review organized — Singularitarian · 2026-10-27
- Music Lenses: ISMIR paper uses sparse autoencoders for steerable concept-level music retrieval — umpedronosapato · 2026-10-27
- Neuronpedia adds 7M+ neurons for Qwen3-32B, Gemma and Llama 3.3 70B to its interpretability encyclopedia — aryaman2020 · 2026-10-27
- Matthew Green hails stunning new ETH result pending Lean proof validity — matthew_d_green · 2026-10-27
- Heuristic for prediction accuracy: a bimodal probability histogram peaking at 0 and 1 — moultano · 2026-10-27
- AI agent autonomously discovers and implements high-order Maxwell solver scheme — burny_tech · 2026-10-27