ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering
Akrin Zheng, Alexander Wu, Alaia Liu
cs.IR, cs.AI, cs.CL
2026-08-11
ScitiX builds an enterprise QA benchmark from an audited truth graph over a real company: 2,341 documents and 907 questions, where L3 latent-organizational items target relations absent from every document; GraphRAG tops out at 36.2% on average and 30.4% stay unanswered even with gold documents supplied.
Enterprise QA is usually framed as retrieval: find relevant internal documents, generate a grounded answer. But enterprise documents are by-products of work, not descriptions written for answering. Which project an engineer belongs to may live as one weekly report saying they worked on a module, other documents placing that module inside a project's workstream, while the sentence "they belong to this project" appears nowhere and must be derived through organizational rules. The paper calls this latent organizational reasoning.
Existing benchmarks miss it because construction runs backwards: fix an answer path, materialize documents along it, and the model merely composes stated facts. Real enterprises offer no such ready-made chain, and a natural information need may be unanswerable outright, because a premise is private, coverage is partial, or the question admits several readings. EntLORE flips the direction: reconstruct an audited enterprise world first, then compile questions from it.
The pipeline has four stages. A provenance-bearing raw graph is rebuilt from routine documents, authoritative organizational tables, and operational logs, with nodes for people, projects, modules, campaigns, systems, and customers, and edges of three kinds: native relations (documented), metadata relations (from authoritative records), and heuristic associations (used only for decoys and candidate discovery, never as gold). A versioned convention library proposes candidate inference edges, for example a containment-transfer rule attributing module-level activity to its parent project; each retained edge records its generating rule, dependency chain, and human authorization, forming the truth graph. An anonymization map consistently renames entities and shifts dates for release, while the truth graph, conventions, and target edges stay withheld. Finally, each question comes from a typed graph operator sampling and executing over the truth graph, computing the answer, proof dependencies, and associated evidence units; a language model only renders the program into employee-like wording and never sees the gold answer.
Three levels follow the target relation's form in the released corpus: L1 answers stated in one document; L2 composed from facts stated across documents; L3 target relations stated nowhere, certified only in the truth graph, recoverable from scattered observations. Pre-release verification checks that the graph program returns valid derivations, set-valued answers are complete under closure, the program has a unique denotation, and for L3 a corpus-wide alias scan confirms absence. Questions requiring an indispensable private premise route to a verified-abstention bank.
The crux of the design: the benchmark holds the truth without releasing it, so evaluated systems see only documents while answers remain verifiable and uncopyable.
907 questions (469 L1, 204 L2, 234 L3) over 2,341 documents and a graph of 1,153 entities with 3,784 typed relations, evaluated across 8 models by 7 access paradigms, 56 configurations. Mean accuracy:
| Access paradigm | L1 | L2 | L3 |
| GraphRAG (corpus-induced) | 63.2 | 45.3 | 36.2 |
| BM25 | 62.4 | 51.4 | 34.0 |
| LLM Wiki | 61.7 | 52.5 | 27.7 |
| Flat RAG | 46.1 | 32.5 | 18.9 |
| Agentic retrieval | 48.5 | 35.6 | 13.5 |
| Gold-document oracle | 87.4 | 93.8 | 69.6 |
Three findings deserve unpacking. Structured access leads but unstably: GraphRAG is best on average, yet per-model it wins five cases and BM25 three, with the gap concentrated in specific relation families, department attribution at 53% for GraphRAG against 2% for BM25, while containment attribution and hierarchy rollup stay under 34% for every deployable method. Dense retrieval collapses across the board: BM25 beats Flat RAG by 16.2, 18.9, and 15.1 points across levels, with about 71% of the gap from document assembly, as lexical matching catches sparse organizational anchors like project aliases, module names, and ticket terms that dense embeddings drop. On the same dense substrate, iterative agentic retrieval (13.5) trails single-pass Flat RAG (18.9), yet given gold document paths the same loop reaches 69.6, so the loop can use evidence and it is the dense backend that fails to fetch it. The sharpest number is the gold-document residual: 12.6% of L1 and 6.2% of L2 remain unanswered, against 30.4% of L3, 4.9 times the L2 residual and 47.7% of the best deployable condition's remaining error. Behavior on failure diverges too: among zero-scored L3 answers, 53% of BM25 outputs explicitly state the fact is absent, versus 6% for GraphRAG.
Release fidelity gets a check: on 176 items that reverse-render cleanly, group-level structure (oracles on top, deployables below) transfers between the private and anonymized worlds, but fine ordering among deployables does not, with deviation concentrated in BM25 and GraphRAG, consistent with lexical sensitivity to an anonymized and translated corpus. The authors self-limit accordingly: group comparisons are valid, fine rankings tentative.
For enterprise RAG deployments, this data explains the familiar field sensation that retrieval succeeds and the answer still fails: the problem is not recall but organizational relations never becoming explicit. The practical hints are concrete, sparse anchors beat dense embeddings on enterprise corpora; offline compilation into a graph or wiki beats adding an agent loop at query time; and systems should treat the organization behind the documents as a first-class inference object that abstains honestly when a relation cannot be derived. The benchmark's code and data are open, and the truth-graph-held-private construction copies to other domains with organizational knowledge, such as healthcare or legal.
One enterprise, one convention library; 234 L3 items spread over five relation families leaves thin per-family counts (department attribution has 42), so family-level conclusions have limited statistical power. The organizational conventions were human-authorized for this enterprise; transferability to other rule systems is unaddressed. Anonymization plus translation breaks fine ordering, which the paper concedes, meaning the two-to-three-point gaps between GraphRAG and BM25 in the table may not be real. The gold-document residual is not a pure derivation-failure measure, still mixing navigation, interaction, and prompt effects, so 30.4% is an upper bound. Free-form answers are scored by a single fixed blind judge; atomic-claim decomposition depends on judge stability, and the appendix replication does not change the single-judge main table. Agentic retrieval ran under caps of 30 iterations, 400,000 tokens, and 900 seconds; whether more budget flips its result is untested.