Grading the Narrators: An Isnad-Rijal Framework for Claim-Level Provenance in Multi-Agent Knowledge Systems
Ali Zahid Raja
cs.AI, cs.MA
2026-07-27
A claim in an LLM knowledge pipeline passes through several transformers. This paper adapts hadith's narrator-chain grading to score each one and cap trust at the weakest link. On 20k physics claims quarantine and corroboration hold, but a weak content critic caps coverage at 4.8%, so the head-to-head win stays unproven.
In an LLM-maintained knowledge base, a single factual claim passes through scraping, extraction, ingestion, and synthesis before it reaches a user. Each step can drop, distort, or invent. Existing provenance work records what happened (execution traces, tool calls, citations), and source-reliability scoring is well established, but neither answers the question a careful reader asks: this specific claim, having passed through these specific transformers, should I trust it?
The author points out that this exact problem was solved once before, with striking rigor, by classical Islamic hadith science.
Hadith scholars had to decide which sayings attributed to the Prophet, transmitted through lengthening chains of narrators, were trustworthy. Their methodology, refined over centuries, has five load-bearing parts: isnad, a complete narrator chain attached to every narration; rijal, a continuously updated graded registry of each narrator's integrity ('adalah) and precision (dabt); weakest-link evaluation, where one weak narrator caps the whole chain; corroboration (mutaba'at), where multiple narrator-disjoint chains can upgrade a claim; and matn criticism, where the content is judged independently of the chain.
The paper transfers this directly. Every transformer (a scraper version, a model at a specific version, a source domain, a human) is a narrator. Grades are domain-conditioned (a model reliable on dates may fail on quantum equations), reset on version bump, and the chain is graded by its weakest link. Independent chains can corroborate an upgrade, and content is criticized separately. A decision matrix of chain grade times content verdict routes each claim to serve, review, or quarantine.
One refinement matters for AI pipelines: destructive transforms (extraction, chunking) can only lose information, so the strict minimum applies; generative transforms (synthesis) can repair or corrupt, so the minimum is bounded and corroboration-gated.
Evaluated on 20,000 atomic claims from real physics textbooks (OpenStax, Crowell), 30% calibration and 70% evaluation, with four narrators carrying designed fault rates (1%, 2%, 15%, 18%).
Weakest-link quarantine worked: every claim whose chain contained a rejected narrator was quarantined (4,057 claims, 29% of the evaluation set), each traceable to the grade that caused it. The grading loop recovered three of four narrator orderings from audit evidence alone, but missed the fourth, the highest-fault (18%) legacy scraper, because it was too rare in the calibration split to be graded at all. Corroboration fired across three corpora, with eight negative controls all correctly declining on the Wikipedia corpus.
Two results are reported as inconclusive. A matched-coverage comparison could not be completed: with the reference content critic, coverage topped out at 4.8%, so no fair head-to-head against the baseline exists. The confidence-gated baseline is uninformative by construction (r = 0.04 with corruption). The framework is open-sourced as pip install isnad, with 157 passing tests.
Multi-agent systems are going mainstream, and whether the chain behind a given claim can be trusted is still largely unanswered. This gives an implementable, traceable framework, and it is unusually honest about where the evidence stops. For anyone building agent systems, knowledge bases, or verifiable AI, the schema and decision matrix are directly reusable.
Set expectations correctly: it proves the mechanism is implementable and behaves correctly where it has evidence, not that it beats current practice at matched coverage. That proof is blocked by the content critic.
The author lists these fully. It targets factual knowledge that can be true or false, not open-ended generation. The content critic is the binding constraint (the reference implementation is word overlap plus a negation heuristic), so coverage is effectively capped at the review budget. Domain-conditioned grading multiplies the cells that need evidence, and a narrator rare in a domain may never get graded, which is exactly why the 18% scraper slipped through. Grading humans turns it into a reputation system over people, with fairness and contestability issues left open. The independence assumption behind corroboration is idealized: two chains sharing no narrator can still fail together if they terminate in the same model family. Nothing has been validated end to end under live production load.