Wilcoxon signed-ranks looks conservative only because of this test dataset
IanArawjo · x · 2026-07-24
The author says the apparent conservatism of Wilcoxon signed-ranks in the plot is an artifact of this test dataset’s judge-score distribution, not a general property of the test. The analysis uses three datasets—WMT translation quality, Chatbot Arena, and App Store reviews—and four judge models: Inkling, Claude Haiku 4.5, Gemma 3 (26B), and GPT-OSS-20b.
The thread is still incomplete: the author says they need to add omnibus tests next, but the current plot already captures the core setup and the caveat about how the statistical result should be interpreted.
Related event: Eliminating LLM Evaluation False Positives with PPI-Corrected Tests(5 posts)→
More from Research
- Paper: data schedules and learning rate schedules should be designed together — alifcoder · 2026-10-27
- Health benchmark study: quality and diversity of eval items beat quantity — keunwoochoi · 2026-10-27
- Across 7 tasks, highest reasoning setting isn't the best for most open models — keunwoochoi · 2026-10-27
- Open Models Match GPT-6 on MedMCQA but Trail 17 Points on Hard Clinical Cases — keunwoochoi · 2026-10-27
- Health-Focused Benchmark Reranks LLMs: Muse Spark Open Jumps from #9 to #1 — keunwoochoi · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27