A new pixel-space benchmark says image generators can reason spatially, but text VLMs still lead
OmniAI-ZJU · hf · 2026-07-24
What the paper proposes
ProVisE (Protocolized Visual Evaluation) is a benchmark-agnostic framework for evaluating spatial cognition in image-generation models using visual answers instead of forcing text or coordinates.
Why it exists
Many spatial benchmarks ask models to output coordinates, multiple-choice options, or text, which creates an interface mismatch for image-generation models that can express answers directly in pixel space.
Main components
- A protocol-constrained visual-answer format that can be parsed into structured predictions
- An Agentic builder that creates and validates protocols for new benchmarks
- SpatialGen-Bench, a diagnostic benchmark with 470 samples across 14 spatial subtasks, four capability levels, and multiple answer forms
Findings
- Image-generation models can be competitive when spatial answers are externalized directly in pixel space.
- Text-output VLMs still have a clear edge in compositional spatial reasoning.
- The framework was validated on six external spatial benchmarks.
Takeaway
The work argues that evaluation format matters: pixel-space expression can expose strengths that text-only tests hide.
More from Multimodal
- Midjourney style code share: --sref 2912175708 — tisch_eins · 2026-09-11
- Astra storyboards plus Minimax H3 per-shot generation boost video success rates — Hailuo_AI · 2026-09-11
- MiniMax H3 MAX nails cooking anime clips: 15-second curry demo with prompts shared — Hailuo_AI · 2026-09-11
- MiniMax Music Production Toolkit 2.5 for ComfyUI adds full mastering chain — Vivid_Promise1700 · 2026-09-11
- New Node Finder for ComfyUI ranks fresh nodes by star velocity and recency — Luke2642 · 2026-09-11
- Using a finisher move on one mosquito with MiniMax H3 MAX — the bug survives — Hailuo_AI · 2026-09-11