Open-source RAG eval tool ships recall, faithfulness and cost metrics with no API key
infrixo · reddit · 2026-11-09
A developer released rag-evaluation-starter, a lightweight Python utility for evaluating RAG pipelines locally against a golden dataset without a full eval platform.
- Metrics: Retrieval Recall@K, answer faithfulness/relevance, exact match, token F1, latency and cost reporting, optional LLM-as-judge
- Core metrics need no API key
- Notable design choice: unanswerable questions are excluded from recall aggregation instead of inflating the score
- Positioned as a point-in-time diagnostic tool, not a replacement for Ragas or LangSmith
More from coding & agent
- S-Agent 2.0 gives coding agents spatial investigation tools, lifting ReVSI by 20.5 points — jiqizhixin · 2026-11-09
- Jane Street's socratic-tutor agent skill teaches by asking questions, not lecturing — intellectronica · 2026-11-09
- Dev building FOSS FL Studio clone: 11-year-old tester finds it intuitive — CtrlAltDwayne · 2026-11-09
- Deep-dive talk on Kamal deployment mechanics, from SSH to proxy handling — js_craft_hq · 2026-11-09
- Harness adds +5.3 points to DeepSeek V4.1 Flash on F1 pit-stop test, at ~19x the cost — AccBalanced · 2026-11-09
- Engineers are the bottleneck: the case for cross-engineer agent-to-agent communication — thomasahle · 2026-11-09