Wilcoxon signed-ranks looks conservative only because of this test dataset
IanArawjo · x · 2026-07-24
The author says the apparent conservatism of Wilcoxon signed-ranks in the plot is an artifact of this test dataset’s judge-score distribution, not a general property of the test. The analysis uses three datasets—WMT translation quality, Chatbot Arena, and App Store reviews—and four judge models: Inkling, Claude Haiku 4.5, Gemma 3 (26B), and GPT-OSS-20b.
The thread is still incomplete: the author says they need to add omnibus tests next, but the current plot already captures the core setup and the caveat about how the statistical result should be interpreted.
Related event: Eliminating LLM Evaluation False Positives with PPI-Corrected Tests(5 posts)→
More from Research
- Nature paper images cellular activity across all organs, revealing body-wide circuits — arjunrajlab · 2026-09-11
- Skyfall GS Uses Flux to Refine Gaussian Splatting, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11
- Could 10k agents discover learning methods beyond backprop, or just tweak existing ones? — SeunghyunSEO7 · 2026-09-11
- Apodex Launches TRACES, First Benchmark for Evaluating 'Discoverative AI' on Real-World Problems — Faheem_uh · 2026-09-11
- TRACES grades the process, not the answer: six-dimension eval for open-ended AI science — Faheem_uh · 2026-09-11
- Apodex launches TRACES, a benchmark grading AI on open-ended discovery instead of known answers — Faheem_uh · 2026-09-11