SIRIN: A Unified Toolkit for Detecting Contextual Hallucinations in Retrieval-Augmented and Memory-Grounded LLM Systems
Julia Belikova, Rauf Parchiev, Mikhail Filimonov, Konstantin Polev, Andrey Savchenko, Maksim Makarenko
cs.AI
2026-07-20
SIRIN unifies probing, judge, and uncertainty hallucination detectors in one toolkit; as a memory-agent gate it lifts accuracy from 62.4% to 79.3% and halves unfaithful answers.
Retrieval-augmented, agentic, and long-term memory LLM systems all share a failure mode: fluent answers that aren't actually backed by the evidence they were given, known as contextual hallucination. Detecting it has split into three camps that don't talk to each other. Probing methods train lightweight classifiers on a model's hidden states. Judge methods use a second model (or API) to fact-check the answer against the context. Uncertainty estimation reads the generator's own confidence signals. Existing open-source tools each cover one camp: LM-Polygraph does uncertainty only, RAGAS and DeepEval are judge-only, LettuceDetect gives span-level output but with a single detector type. Nothing lets you mix signals from all three families or compare them side by side without rewriting a pipeline for each.
SIRIN wraps probing, judge, and uncertainty detectors, plus a pre-generation task called query answerability (is the context even sufficient to answer this?), behind one interface, one config system, and one evaluation pipeline. Every detector emits the same report format: a calibrated score, a predicted label, and optional evidence such as unsupported spans or a judge's rationale. Backends are swappable: Transformers for white-box access to hidden states, vLLM for high-throughput generation and log-probability scoring, and OpenAI-compatible endpoints for black-box judges. A shared forward pass extracts multiple feature families at once, so running several probing signals together costs close to one model call instead of one call per signal. A Streamlit web UI lets a user paste a context-query-answer triple, pick a detector combination, and immediately see a hallucination score, highlighted unsupported spans, and a side-by-side comparison of detectors.
On the three RAGTruth splits (QA, summarization, data-to-text), a LoRA-tuned Qwen3-4B judge is the strongest option when labels are plentiful, reaching 91.8-93.5 AUROC, 5.5-9.0 points above the best probe on the same split. A zero-shot GPT-5.4-mini judge is close behind. The picture flips on SQuAD 2.0, where labels are scarcer: a lightweight attention-pooling probe trails the LoRA judge by only a few points (78.4 vs. 87.0 AUROC), making it the more annotation-efficient choice. Judges also win at span-level localization on PsiloQA (74.8 F1, 69.3 IoU), while zero-shot span tagging collapses entirely. The most convincing result comes from wiring SIRIN into the SimpleMem long-term memory system as a selective-prediction layer: stacking an answerability gate before generation and a faithfulness gate after it lifts served accuracy on LongMemEval from an ungated 62.4% to 79.3%, while cutting strict unfaithfulness from 20.2% to 9.8%. The same two-gate design carries over to two other memory frameworks, Mem0 (+13.2 points) and LightMem (+14.3 points).
This isn't a new detection algorithm. It's a trust layer that drops into an existing RAG or agent-memory pipeline without touching application code: two gates around the retrieval stage, one checking whether the evidence is sufficient before generation, one checking whether the answer is grounded after it. For teams already running RAG or long-term memory agents, the main value is skipping the engineering work of stitching together three separate detection frameworks, plus a clear rule of thumb for which detector family to use when: judges when labels are plentiful, probes when they're scarce or compute is tight.
Both probing and judge detectors are trained and validated on English only; porting to other languages means re-annotating and re-tuning from scratch. The memory-gating experiments use a single generator backbone (Qwen3.5-35B-A3B) paired with one probe configuration and one judge; whether the gates transfer to a very different backbone without refitting is untested. The web UI also caps how many model configurations can stay warm at once (6-8 on one A100), so live comparison of many detector combinations needs extra hardware. The Mem0 and LightMem results use a revised relabeling protocol that isn't identical to the main SimpleMem table, so cross-framework comparisons should be read with that caveat.