Wilcoxon Signed Rank Outperforms McNemar for Binary Eval Data
IanArawjo · x · 2026-08-31
A finding from the evalstats paper holds up: for pairwise p-value tests on binary data, Wilcoxon signed rank is the recommended method. It outperforms the McNemar mid-p test on power without degrading Type I error. While binary data has many ties—suggesting Wilcoxon should reduce to the ultra-conservative Sign Test—scipy's implementation includes tie-variance correction, yielding this robust and surprising result.
Related event: Wilcoxon Signed Rank Test Beats McNemar for Binary Evals(2 posts)→
More from Research
- Handwritten Code vs. AI-Generated Code in Generative Art — adamho · 2026-09-01
- UChicago Booth to host World Models workshop in 2026 — ethayarajh · 2026-09-01
- Agents Develop 'Classical Chinese' to Compress Long-Chain Task Context — wzenus · 2026-09-01
- Paper: Project MUSE - Getting to Know Worst-Case Confounders — RexDouglass · 2026-09-01
- 4D Gaussian Splatting Lacks Standard, Tools Still Experimental — Michael_Moroz_ · 2026-09-01
- Microsoft releases GigaPath-Flash: Efficient pathology models cut compute by 50x — Microsoft Research · 2026-09-01