Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability
Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, Hwanjun Song
COLM 2026
cs.AI
2026-07-31
Q-CARE decomposes queries and answers, scoring RAG on coverage and verifiability without reference answers, and matches human judgment best across eight datasets.
RAG systems can fail at retrieval, generation, or the seam between them, which makes evaluation hard. Existing frameworks are stuck on a structural trade-off. RAGChecker and RAGEval give claim-level diagnostics but lean on reference answers or human-labeled chunks, so they only fit close-ended queries with a single correct answer. ARES, RAGAs, and DoRAG drop the reference requirement and handle open-ended queries, but at a much coarser granularity. No method is fine-grained, reference-free, and works on both query types at once.
The deeper issue is that prior work treats correctness as isolated checks (is this chunk relevant, does this claim match a reference) instead of asking whether the query's intent was satisfied, a question that does not depend on query type.
Q-CARE (COLM 2026, KAIST) redefines a correct answer as two judgments that hold for any query: coverage (the answer addresses every information need implied by the query) and verifiability (every statement in the answer is supported by retrieved evidence). Translating any query into these two signals makes the metric query-agnostic.
It runs in three stages:
The authors built a human-annotated benchmark across eight datasets, 567 sub-queries and 7,907 claims, with inter-annotator Fleiss' kappa of 0.80 to 0.89. On Pearson correlation with human judgment:
| Dimension (open-ended) | Q-CARE | Best baseline (RAGChecker) |
| Completeness | 0.47 | 0.17 |
| Verifiability | 0.69 | 0.56 |
Q-CARE is highest on every dimension and both query types, while RAGAs and RAGChecker lose correlation when moved from close-ended to open-ended. The retrieval story is sharper: ranking 21 retrievers with traditional Precision@k gives a cross-query-type rank consistency (Spearman) of only 0.55, but the coverage-aware C-Prec@k lifts it to 0.89. Without modeling query coverage, system-level rankings can be misleading end to end.
Latency is comparable to RAGChecker (about 55 s close-ended and 98 s open-ended per query) and far below DoRAG's 390 to 466 s. Stronger backbones correlate better, but an 8B model is nearly unusable on open-ended queries.
For anyone evaluating or selecting RAG systems, this is one of the few frameworks that diagnoses both retrieval sufficiency and answer accuracy without collecting reference answers. Reference-free scoring matters most for open-ended, long-form answers, where building references is itself impractical. It also tells you whether a failure is retrieval or generation, instead of returning a single score.
The ceiling is set by the judge LLM: an 8B model is close to useless on open-ended queries, so Q-CARE is only reliable with a strong backbone. Each query triggers dozens of LLM calls (55 to 98 s), parallelizable but not cheap. The authors note binary yes/no alignment actually beats three-level scoring on open-ended queries, so fine-grained is not always better. The human-correlation set is 320 queries, a modest sample.