ExtractBench Reveals VLMs and Coding Agents Fail at Document Grounding
llama_index · x · 2026-08-17
LlamaIndex released ExtractBench, a benchmark strictly evaluating document extraction grounding, requiring correct citations with correct values (IoU 0.5). Key findings:
- VLMs and coding agents fail completely, scoring zero at both citation levels.
- Current systems struggle with word-level precision (F1 < 50%) and suffer from performance collapse on long documents.
- LlamaExtract leads with 84.9% page-level and 46.4% word-level scores, maintaining 87.1% performance on long documents where others hit zero.
Related event: LlamaIndex Releases ExtractBench, VLMs Fail at Attribution(2 posts)→
More from coding & agent
- Inflight launches: a feedback layer that turns prototype comments into Claude tasks — soleio · 2026-10-16
- Apple code signing drags for hours; official advice: just wait a week — pjausovec · 2026-10-16
- Hermes: a 4B model with 8K context got worse with more test-time compute, pointing to contextual reasoning — mangahomanga · 2026-10-16
- Flux 3 Klein's Default Templates Are Broken Without Its Prompt Upscaler — StrictMove5839 · 2026-10-16
- 21-person team ships hundreds of PRs weekly with AI writing 97% of the code — rseroter · 2026-10-16
- ClosedMCP: Set Up MCP Servers Once, Share Them Across Your Whole Team — Ok_Sprinkles1301 · 2026-10-16