PixelBench: testing multimodal models pixel-by-pixel instead of with QA

bdsqlsz · x · 2026-11-04

Researchers introduce PixelBench, arguing the QA-based evaluation era for multimodal models is over. QA only tests high-level text reasoning, while SVG and CAD rely on symbolic shapes rendered by external engines. PixelBench is a direct text-to-pixel probe where LLMs and VLMs generate and understand images pixel by pixel (e.g., 16×16 pixel art of mushrooms, owls, cherries, turtles). Project page: haoquanzhang.github.io/pixelbench

Original post →

PixelBench: testing multimodal models pixel-by-pixel instead of with QA

More from Multimodal