EMNLP paper introduces CAP, an 8-class semantic correctness metric for open QA
preslav_nakov · x · 2026-10-26
An EMNLP 2026 Findings paper tackles a key gap in open-ended QA evaluation: correctness isn't binary—answers can be partial, verbose-but-correct, overinclusive, or contaminated with hallucinated facts, and existing judgment/similarity metrics collapse these distinctions. Contributions: an ordered 8-class semantic correctness taxonomy (Exact, Equivalent, Alternative-correct, Overinclusive-valid, Partial, Overinclusive-invalid, Invalid, Contradictory) with a monotonicity criterion; two datasets—CAP-Correctness (8.8k examples, 1.6k human-labeled) and CAP-Statements (11k QA-to-statement examples for NLI training); and CAP (Context-Aware Precision), a reference-based metric scoring question-conditioned statements via bidirectional NLI. Under a monotonicity protocol, CAP scores 77.70 vs 56.18 for BERTScore, and popular evaluators frequently violate the intended correctness ordering. Team from Sofia University, ETH Zürich and MBZUAI; code and data released.
More from Research
- Open textbook on robot learning launches with chapters on control and imitation to VLA — GlenBerseth · 2026-10-26
- A $5M plan to test whether 40Hz music could treat or prevent Alzheimer's — jajoosam · 2026-10-26
- Judea Pearl Notes His Book 'Causality' Has 'Infinite Lifespan' With 5,000+ Citations — yudapearl · 2026-10-26
- Animating brain tractography with a traveling light pulse: the simple frame-length formula — Dr_Alex_Crimi · 2026-10-26
- Clinicians over-rely on wrong AI findings and under-rely on correct ones, Lancet study — zakkohane · 2026-10-26
- Depth-Grown and Looped LMs Learn In-Context Learning Earlier Than Dense Models — omarsar0 · 2026-10-26