A new pixel-space benchmark says image generators can reason spatially, but text VLMs still lead
OmniAI-ZJU · hf · 2026-07-24
What the paper proposes
ProVisE (Protocolized Visual Evaluation) is a benchmark-agnostic framework for evaluating spatial cognition in image-generation models using visual answers instead of forcing text or coordinates.
Why it exists
Many spatial benchmarks ask models to output coordinates, multiple-choice options, or text, which creates an interface mismatch for image-generation models that can express answers directly in pixel space.
Main components
- A protocol-constrained visual-answer format that can be parsed into structured predictions
- An Agentic builder that creates and validates protocols for new benchmarks
- SpatialGen-Bench, a diagnostic benchmark with 470 samples across 14 spatial subtasks, four capability levels, and multiple answer forms
Findings
- Image-generation models can be competitive when spatial answers are externalized directly in pixel space.
- Text-output VLMs still have a clear edge in compositional spatial reasoning.
- The framework was validated on six external spatial benchmarks.
Takeaway
The work argues that evaluation format matters: pixel-space expression can expose strengths that text-only tests hide.
More from Multimodal
- Adobe Firefly video editor walkthrough: one-sentence Quick Cut turns 32s into 20s — LudovicCreator · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26
- Claude Can Replace Your YouTube Thumbnail Designer in 10 Minutes — thedankoe · 2026-10-26
- Multimodal AI Is Not Useless: A Practitioner Pushes Back on Growing Skepticism — thoma_gu · 2026-10-26
- One prompt, full Blender animation: dev says he never touched Blender once — iamfakhrealam · 2026-10-26
- Dense instance segmentation via Muse Wonder + SAM 3.1 on Meta's Model API — nikhilaravi · 2026-10-26