Unanswerable questions inflate Recall@K: dev open-sources RAG evaluator to fix it
infrixo · reddit · 2026-10-31
The author spotted a metric pitfall while reviewing a RAG evaluation dataset: if a question has no relevant documents, counting it as a perfect retrieval result inflates Recall@K.
- In a small 8-question example: naive average Recall@K is 0.75, dropping to 0.714 after excluding the unanswerable question — the retriever didn't improve, the measurement was misleading.
- They maintain an open-source RAG evaluation starter (infrixo-systems/rag-evaluation-starter) with retrieval metrics, answer evaluation, and optional LLM-based judging; core metrics run without an API key.
- They're asking how others handle unanswerable questions: exclude from retrieval recall and evaluate separately?
More from coding & agent
- Devs Rageward-Hacked: LLMs Bias Toward New Code, Wreck Real Projects — yacineMTB · 2026-10-31
- Playable Environments: Millisecond Cloud Branching as the Next Test Bed for Agents — matei_zaharia · 2026-10-31
- Using GitHub Copilot to Figure Out a Device's Pin Layout From a Photo — DanWahlin · 2026-10-31
- OpenAI's MCP Review Rules: Every Tool Needs Annotations, but Hints Aren't Security — shoumikgoswami · 2026-10-31
- Gabriel Chua finds Codex quietly running end-to-end tests on its own Cover Studio app — gabrielchua · 2026-10-31
- App Store Connect CLI 6.2.0 adds enterprise API keys, an 8.1k-star tool for iOS release automation — rudrank · 2026-10-31