Wilcoxon signed-ranks looks conservative only because of this test dataset
IanArawjo · x · 2026-07-24
The author says the apparent conservatism of Wilcoxon signed-ranks in the plot is an artifact of this test dataset’s judge-score distribution, not a general property of the test. The analysis uses three datasets—WMT translation quality, Chatbot Arena, and App Store reviews—and four judge models: Inkling, Claude Haiku 4.5, Gemma 3 (26B), and GPT-OSS-20b.
The thread is still incomplete: the author says they need to add omnibus tests next, but the current plot already captures the core setup and the caveat about how the statistical result should be interpreted.
Related event: Eliminating LLM Evaluation False Positives with PPI-Corrected Tests(5 posts)→
More from Research
- Kimi K3 may be strong on cyber, but token efficiency keeps it off UK AISIS — teortaxesTex · 2026-07-27
- ARC AGI 3 should have stayed private, with no examples or public dataset — flowersslop · 2026-07-27
- ExploitGym may have only 60–70% solvable tasks, fueling the OpenAI cheating debate — max_paperclips · 2026-07-27
- RTX 5090 local tests show Qwen Q6 can drop to 15 tok/s at 80k context — LFAdvice7984 · 2026-07-27
- Noahpinion quotes Chollet: intelligence may hit a hard ceiling — binarybits · 2026-07-27
- Paper argues graph topology can become the core operating system for AI agents — theomitsa · 2026-07-27