Open-source RAG evaluator exposes how unanswerable questions inflate Recall@K
infrixo · reddit · 2026-11-03
The author found a subtle flaw in RAG evaluation: questions with no relevant documents get treated as perfect retrieval results, inflating Recall@K. In an 8-question example, naive averaging gave 0.75, but excluding the unanswerable question dropped it to 0.714 — the retriever hadn't improved, only the metric was misleading.
They released an open-source RAG evaluation starter (infrixo-systems/rag-evaluation-starter) covering retrieval metrics, answer evaluation, and optional LLM-based judging, with core metrics runnable without an API key. They invite discussion on whether others exclude unanswerable questions from retrieval recall and evaluate them separately.
More from Research
- Princeton's RAVEN stores raw visual embeddings as robot memory, beating caption-based baselines — shahdhruv_ · 2026-11-03
- Chemist Finds AI Reproduced Months of His 2021 Reaction Calculations in Minutes — CatAstro_Piyush · 2026-11-03
- Prefrontal open-sources LSA: 72.4x faster attention prefill at 10M tokens, accuracy on par — jcrwhittington · 2026-11-03
- Meta paper: routing agent memory by task type beats shared history, cuts API cost 65-72% — rohanpaul_ai · 2026-11-03
- Meta FAIR paper: research agents work best with humans, hack detection still weak — armand_ruiz · 2026-11-03
- Stanford Re-tests 50 Widely Cited LLM Findings: Most Expire or Fade Within 15 Months — ziv_ravid · 2026-11-03