RECTIFY: An Interactive Workbench for Post-Evaluation RAG Diagnosis, Repair, and Verification
Keerthana Murugaraj, Salima Lamsiyah, Martin Theobald
cs.SE
2026-09-15
RECTIFY routes evaluated RAG cases into 4 families and 23 repair slices; on a 100-question synthetic benchmark it cuts repair decisions by about 97%, down to 2-3 cards.
RAG evaluators can flag weak retrieval, shaky grounding, incomplete answers, and unsupported generation. They rarely tell a developer which component to change next. Treating every failed case as its own bug makes repair slow and hard to audit. RAGXplain writes per-case advice; Doctor-RAG localizes faults in agent traces. Neither turns recurring failures into one reviewable repair card and then checks the patch in a sandbox.
RECTIFY, from the University of Luxembourg, fills that post-evaluation gap: evaluated cases become an auditable repair workflow, and a human still hits approve.
Each case is normalized into a shared schema: question, answer, retrieved contexts, optional gold, scores, and diagnostic fields. The primary evaluator is RAGVue, from the same authors, covering retrieval quality, relevance and completeness, strict faithfulness, and calibration.
Two pre-filters run first. Unanswerable items with a correct refusal drop out. Answerable items whose output is close to gold and grounded in retrieved context drop out too. Remaining cases fall into four macro families in priority order: abstention, retrieval, grounding, generation. Abstention comes first because a confident unsupported answer is worse than noisy retrieval. Retrieval comes before grounding because faithfulness is hard to read on weak evidence. Inside a family, cases land in one of 23 slices, such as R2 noisy retrieval, R5 multi-part under-retrieval, or S3 underused evidence. Each slice fills a template repair card: target stage, proposed change, expected benefit, tradeoff, and scope. Developers edit, approve, or reject; the decision is logged. An optional sandbox applies the patch to the affected cases and reports improved, unchanged, or regressed.
Routing is deterministic. It does not ask another generator to invent a root cause.
The benchmark is 100 synthetic questions over 30 short documents about 10 fictional companies: 84 answerable, 16 unanswerable. Three retrievers share a local Mistral-7B generator, which also judges 12 RAGVue metrics: BM25, dense all-MiniLM-L6-v2, and hybrid.
| Setup | Final repair agenda | Dominant slices |
| BM25 | 16 | R2 noisy retrieval 7, R5=3, S3=6 |
| Dense | 5 | R2=0, R5=4, S3=1 |
| Hybrid | 6 | R2=0, R5=1, S3=5 |
Strict faithfulness moves from 0.627 on BM25 to 0.745 dense and 0.741 hybrid. Answer completeness moves from 0.391 to 0.474/0.465. 81 questions need no repair under any retriever. A raw metric scan inspects 74-89 cases; pre-filters leave 5-16; slices collapse those into 2-3 cards, a 97% cut in decision count. In the sandbox, reranking all 7 BM25 R2 cases helps all 7. Raising top-k from 3 to 6 on R5 helps 5 of 8, leaves 2 unchanged, and regresses 1.
Teams already running RAGAS, RAGChecker, or RAGVue often lack a way to turn scores into an approvable choice: rerank, raise top-k, or tighten the prompt. RECTIFY changes the debugging unit from a single case to a slice-level patch and keeps a human in the loop. It is an engineering workbench, not a new generator. The synthetic corpus buys control over gold documents and unanswerable items; enterprise knowledge bases still need their own test.
The authors call this a local interactive prototype, not a production optimizer. The 23-slice taxonomy is fixed and must grow with new architectures. Without gold labels, annotation-based pre-filters turn off and diagnosis rests on evaluator signals. Generator and judge are the same Mistral-7B, so scores can flatter each other. The sandbox only shows two slices, and R5 already regresses once, which is the point of treating cards as hypotheses. Some appendix metric names scrambled in HTML conversion; the body tables are the source of record.