ExtractBench Reveals VLMs and Coding Agents Fail at Document Grounding
llama_index · x · 2026-08-17
LlamaIndex released ExtractBench, a benchmark strictly evaluating document extraction grounding, requiring correct citations with correct values (IoU 0.5). Key findings:
- VLMs and coding agents fail completely, scoring zero at both citation levels.
- Current systems struggle with word-level precision (F1 < 50%) and suffer from performance collapse on long documents.
- LlamaExtract leads with 84.9% page-level and 46.4% word-level scores, maintaining 87.1% performance on long documents where others hit zero.
Related event: LlamaIndex Releases ExtractBench, VLMs Fail at Attribution(2 posts)→
More from coding & agent
- Gemini 3.7 praised as 'too fast and too good' for agent workloads — DynamicWebPaige · 2026-08-18
- Peter Yang's tutorial: building 5 personal agents with Grok Bot — petergyang · 2026-08-18
- Vorflux opens sign-ups: cloud platform turns a sentence into a merged PR — testingcatalog · 2026-08-18
- cramforce: 'Agents are tools in a loop. Factories are agents in a loop.' — cramforce · 2026-08-18
- 'Here's constitution.md, go nuts': onboarding in the agent era — marilynika · 2026-08-18
- Autonomous OS open-sourced: Android for robots — dee_hw · 2026-08-18