RECTIFY: Open-Source Workbench Turns RAG Evaluation Failures into Verifiable Repair Cards
_reachsumit · x · 2026-09-16
- The paper RECTIFY introduces an open-source Streamlit workbench addressing a gap in RAG tooling: evaluators can flag failures (weak retrieval, poor grounding, unsupported generation) but rarely tell developers what to fix next.
- RECTIFY filters out cases needing no repair, routes remaining failures into actionable families and fine-grained slices, and generates editable repair cards that developers can approve, reject, or verify via sandbox reruns.
- On a controlled benchmark it surfaces interpretable failure profiles: BM25 mainly triggers noisy-retrieval repairs, while dense and hybrid retrieval leave smaller sets of multi-part underretrieval and underused-evidence cases.
- Additional analysis shows pre-filtering cuts unnecessary repair candidates, and slice-level routing yields more targeted repair cards than broad family-level diagnosis. The workbench is publicly available as open source.
More from coding & agent
- TypeSafe's Jev Cut Grok Build Costs 22-40% on the Same Tasks — Daniel_Farinax · 2026-09-21
- jev-skill-suggester: a lightweight pre-execution skill router stops agents picking wrong tools — udmrzn · 2026-09-21
- OpenClaw adds FaceTime: your agent can now call you, and answer your calls — steipete · 2026-09-21
- Reverse-engineering Instinct's memory: beautifully simple, replicable in 60 lines — julianweisser · 2026-09-21
- 3 Weeks of Logs Show 40% of Messages to Coding Agents Are Overhead — Lucky-Group9525 · 2026-09-21
- Humanizer Academic: a one-file skill that de-AI-fies academic medical papers — udmrzn · 2026-09-21