LlamaIndex launches ExtractBench Pro: long-tail document extraction that stumps frontier models
llama_index · x · 2026-10-20
Jerry Liu and LlamaIndex introduced ExtractBench Pro, a benchmark for real-world document extraction that even frontier models (Astra/Opus with max thinking) struggle with.
Key points:
- Existing extraction benchmarks are saturated — their cheapest modes already score 93%+ — but a massive long tail of use cases (trial balances, well logs, loss runs, site plan reviews) remains unsolved on price-performance.
- Hard document automation requires deep visual perception/OCR of complex entities plus a reasoning trace reconciling contradictory sources, as shown in a water loss audit example requiring the main diagram and appendix date changes.
- The team frames complex extraction as a microcosm of professional PDF work and sees room for specialized models pushing cost/accuracy frontiers. Announced alongside Extract v2.5 document extraction agents.
More from coding & agent
- Microsoft's TokenEconomizer cuts coding agents' recurring token costs with a black-box controller — MicrosoftResearch · 2026-10-20
- User: Codex burns half of the day's usage limits within an hour after banked reset — ssh4net · 2026-10-20
- Berth: a Go binary that time-shares one GPU between AI engines and games via leases — Distinct-Pie2389 · 2026-10-20
- Building Tools Inside Tools to Build the Tool: Game Dev's Inception — eschadiol · 2026-10-20
- reuF2F: 244 open math problems formalized in Lean, best coding agent solves only ~40% — AkariAsai · 2026-10-20
- When should an agent combine tool readings without a shared snapshot? — InformationLive493 · 2026-10-20