SynthDocBench Reveals Doc Understanding Failures
ServiceNow-AI · hf · 2026-07-15
ServiceNow-AI introduces SynthDocBench, a fully synthetic benchmark for long-context visual document understanding.
This benchmark features independent control over:
- Document length
- Layout structure
- Modality combinations
- Question types
The authors used an LLM pipeline to automatically generate documents, introducing 40% random occlusion across six layout prototypes to prevent models from exploiting spurious correlations. Evaluations on 7 frontier VLMs revealed several issues that existing benchmarks fail to expose:
- Performance drops significantly as document length increases.
- Most models are highly sensitive to the middle sections of documents, with the middle third being the hardest to process.
- Chart understanding systematically fails in long-document scenarios.
More from Multimodal
- HeyGen adds a media-sourcing skill for coding agents with 75k images and 10k tracks — HeyGen · 2026-07-22
- Reddit user seeks ComfyUI NSFW text-to-image and image-to-video workflows under 20 GB VRAM — hobbyist2020 · 2026-07-22
- Krea 2 users recommend a two-pass Clownshark sampler setup for sharper image details — listopalafoto · 2026-07-22
- Gemini Omni Flash turns a boat cabin into a cave in Flow by Google — chrisfirst · 2026-07-22
- A simple workflow to turn a photo into an image prompt using Gemini, Grok, or GPT Image — harshitagu72595 · 2026-07-22
- A Reddit user proposes a consistency LoRA to keep anime and game scenes visually stable — ThirdWorldBoy21 · 2026-07-22