ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction
Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo
cs.AI
2026-08-01
ExtractBench benchmarks 14 extraction systems on 370 enterprise documents and 4,869 pages. VLMs ace short docs but collapse on long ones and 1000-row tables, while source grounding stays unsolved.
Enterprises turn business documents (invoices, insurance claims, government forms, financial filings) into structured data, a job once done by hand. Hand it to an LLM agent and a new hard requirement appears: it cannot just return JSON. Every value has to be traceable to the exact page and word box it came from, or the result is not auditable. Existing benchmarks measure at most one of value accuracy, long-document record completeness, grounding, and cost. None put them together. ExtractBench fills that gap.
Short documents are easy for everyone. What separates systems is whether they silently truncate thousand-row record lists on long documents, and whether they can anchor each number back to the source. Those are the two things no prior benchmark tested.
The task is simple to state: given a document and a user-authored JSON Schema (field names, types, descriptions), return schema-valid JSON with a source page and word-level bounding box per value.
The dataset spans 370 enterprise documents, 4,869 pages, 8 industries, and 67 document types, tagged along 13 challenge axes. Ground truth is built three ways, each for a reason. Real documents are extracted by an ensemble of independent systems; consensus becomes truth, and only disagreements go to human adjudication, because a single extractor would bake its own biases into the answer key. Very long lists (thousands of rows) are impossible to label by hand, so the data is generated first and rendered into PDF afterward, making ground truth exact by construction. Scanned and handwritten forms are ambiguous, so a human verifies every field and places boxes; 84% of verified fields carry human-placed boxes.
Three metric families: order-insensitive value F1 (records aligned by the Hungarian algorithm after date normalization and whitespace collapsing, missing fields scored as explicit null), word-level grounding F1, and page-level grounding F1. Grounding is strict: a field counts as grounded only when the value is correct and the predicted box overlaps the accepted box at IoU at least 0.5.
Fourteen systems competed: 8 VLMs, 2 coding agents (Claude Code Opus 4.8, Codex GPT-5.5), and 4 specialized extraction APIs. On overall value F1, LlamaExtract Agentic Plus led at 95.6%, Codex GPT-5.5 second at 93.6%, Reducto Deep Extract third at 90.4%. Cost tells the real story:
| System | Overall F1 | Cost/page |
| LlamaExtract Cost-Effective | 86.8% | 1.0¢ |
| LlamaExtract Agentic | 89.5% | 3.1¢ |
| LlamaExtract Agentic Plus | 95.6% | 8.1¢ |
| Codex GPT-5.5 | 93.6% | 27.8¢ |
| Claude Code Opus 4.8 | 87.1% | 15¢ |
Agentic Plus matches or beats the coding agents at less than a third of Codex's cost.
Document length and enormous tables split the field. By length, Gemini 3.5 Flash falls from 87.9% on short documents to 27.9% on long ones, and Qwen3.6 to 26.8%. Tables beyond 1,000 rows (S4) are brutal: every VLM sits under 10%, the worst near 1%, while Agentic Plus holds 95.9%, Reducto 95.3%, and Claude Code 87.8%. The paper is blunt about why: most systems cannot work through a long document in a single pass, and those without an iteration strategy stop early and drop the remaining records.
Grounding is the other open problem. VLMs and coding agents return no evidence by default, scoring zero at both levels. Among systems that do return evidence, page-level is tolerable (Agentic Plus 84.9%), but word-level collapses, with the best system at 46.4%. Pinning a number to the exact word is much harder than finding the right page.
If an enterprise only needs short documents with correct fields, a closed VLM is already cheap and good enough. The moment long lists, enormous tables, or audit-grade traceability enter the picture, general VLMs are effectively unusable today, and you need a specialized system with pagination and grounding. The benchmark's value is not picking a winner but providing a ruler that separates toy short-document scores from enterprise-ready performance.
For anyone building RAG or document agents, it quantifies a point worth repeating: a big context window does not mean a model can handle long documents. The same model can drop 60 points from short to long.
The grounding ceiling is so low that even the top system's word-level F1 sits under 50%, which suggests the metric may be uniformly too hard for current systems and might be better reported in tiers. Costs are estimated from public pricing; negotiated and self-hosted costs can differ widely. The benchmark authors are LlamaIndex, which makes LlamaExtract, the product that sweeps first place on every metric, an obvious conflict of interest to discount. Synthetic long lists guarantee exact ground truth but may not match the messiest real-world scans in layout complexity.