Before Reasoning Fails: Pre-Evidence Procedural Failures in Agentic RAG
Daeyoung Roh, Donghee Han
cs.AI
2026-08-03
Across 12,000 RAG trajectories, agents often answer without reading retrieved evidence; a weight-free runtime gate forcing read-before-answer recovers 14.9 to 19.9 accuracy points.
Agentic RAG (retrieval-augmented generation with tool calls) differs from fixed-context RAG: the agent decides when to search, which result to read, when to stop, and when to answer. That freedom creates a procedural risk: the agent may retrieve candidate snippets but finalize an answer without inspecting them. The failure happens before evidence-conditioned reasoning is ever tested.
The authors argue this failure should be separated from "read the gold evidence but still reasoned wrong." Folding both into generic wrong answers hides a failure source that is cheap to fix.
The framework splits wrong answers into buckets. One is discipline failure, with three subtypes: no-read final (readcount is zero when the answer is emitted), snippet-only final (the answer entity appears only in search snippets, never in a read chunk), and low-evidence final (question entities covered at less than 80% in read chunks). The other is post-gold-read failure: retrieval succeeded, the agent read at least one gold-supporting chunk, and the answer is still wrong.
The core intervention is Read-Gate, a runtime, environment-level invariant: if the agent emits a final answer with readcount zero, the environment rejects the action and returns a corrective hint telling it to call read on a promising chunk first. It changes no model weights, no retrieval, no index, no judge, no reasoning budget; it only enforces "read before you answer."
Experiments run on HotpotQA, 2WikiMultiHopQA, and MuSiQue, 1,000 examples per dataset-condition pair, for 12,000 OpenAI trajectories total, with gpt-4o-mini and gpt-5-mini as primary controllers.
First, do the two failure types overlap? Across 3,807 wrong cases, the both-trigger rate sits at 11.2% to 13.1% (stable across regex and spaCy extractors), far below the threshold at which they would collapse into one. These are two distinct failure regimes.
Read-Gate's effect is sharpest on the subset that would otherwise skip reading.
| Dataset | Zero-read LLM-Acc (original, then forced-read) | Gain |
| HotpotQA | 58.1 to 73.0 | +14.9 |
| 2WikiMultiHopQA | 42.1 to 62.1 | +19.9 |
| MuSiQue | 22.5 to 37.4 | +14.9 |
On full minimal-reasoning cells, the gain is +3.2 to +9.4 points. Paired McNemar p-values are all below 1e-4.
One control matters: injecting chunk text into context without a read action (ctx-inject) fails to replicate Read-Gate's gains, and is even negative on 2Wiki (down 7.4 points). What helps is forcing the agent to read, not merely handing it more text.
The most counterintuitive result comes from Gemini 2.5 Flash: adding a 1024-token thinking budget raises zero-read finalization rather than lowering it (HotpotQA +5.7 pp, 2Wiki +24.8, MuSiQue +42.6), and accuracy drops. Hidden deliberation and actually inspecting evidence are different things and can move independently.
This makes a plain point clearly: evidence gathering is a trajectory-level control problem to be evaluated and fixed separately from answer-side reasoning. A small inference-time gate that touches no weights and no retrieval recovers 14.9 to 19.9 points on the zero-read subset, which is useful for cost-sensitive deployment. It also closes 20% to 24% of the accuracy gap from minimal to medium reasoning, at the cost of a few extra reads.
The counterintuitive finding is worth remembering: more thinking budget does not necessarily make an agent more disciplined. Sometimes it makes the agent more inclined to skip evidence and answer directly.
The authors note the scope: only cost-sensitive controllers (gpt-4o-mini, gpt-5-mini) are covered; Gemini is used only as an external diagnostic. The framework requires discrete search/read/final tool actions and does not fit agents that interleave retrieval and generation implicitly. All three datasets are English Wikipedia-style multi-hop QA with explicit evidence chains, which is exactly what makes discipline failures visible; other domains would need recalibrated coverage thresholds and gate behavior.
Read-Gate is also not a retrieval-quality fix. It assumes search often returns useful candidates, and its cost is more reads and more retrieved tokens per question, so it should sit alongside retrieval monitoring and answer-side verification.
One caveat: procedural failure is especially visible on these multi-hop benchmarks, perhaps partly because answer entities concentrate in a few chunks, so "did it read" is clean to judge. Whether Read-Gate gains as much in domains with more dispersed answers and fragmented evidence is not shown here.