EMNLP 2026 paper uses prediction-powered inference to quantify how much human annotation LLM judges save
shi_weiyan · x · 2026-10-22
An EMNLP 2026 main conference paper, "Which Metrics Save the Most Human Annotation?", tackles the trade-off between costly human evaluations and biased LLM judges.
Building on prediction-powered inference (PPI), which combines small-scale human judgments with large-scale LLM judgments to correct judge bias, the paper contributes:
- A nonparametric permutation test replacing PPI's default parametric Z-test, which is unreliable at small sample sizes
- Evidence that unpaired designs are less statistically efficient than paired ones
- A new meta-metric measuring how many human annotations an LLM judge (autorater) can save, expressed as a "saving ratio" to guide metric selection
The author presents a poster on Oct 27, 11:00 AM–12:30 PM, Hall H, at EMNLP 2026.
More from Research
- Mirror hits Minecraft milestones via pure self-play, 9x DreamerV3's iron pickaxe rate without rewards — vincesitzmann · 2026-10-22
- Open textbook explains when controller stability is topologically impossible — burkov · 2026-10-22
- Toward inverse symbolic learning: back out physics equations from learned deep-net world models — bradneuberg · 2026-10-22
- SierpinskiCam (NeurIPS 2026) lets you re-shoot filmed video from new camera paths — taiyasaki · 2026-10-22
- Endless Powder streams an infinite ski mountain with a bounded 64 MiB working set — sterlingcrispin · 2026-10-22
- Retracting old papers: 'Is it still load-bearing?' beats 'Is it too old?' — iskander · 2026-10-22