Image-Space Rule Discovery
Misora Sugiyama, Toya Oyama, Hirokatsu Kataoka
cs.CV
2026-08-01
WISRD turns visual-reasoning problems into 1024x1024 worksheets that image-editing models must read, solve, and write back into the image; the best model scores 48.7%, InstructPix2Pix zero, humans 96%.
Image-editing models such as Nano Banana Pro, FLUX.2, and Qwen-Image-Edit are good at making pictures look different. But do they understand what is in the picture? This paper skips "does the edit look nice" and asks something harder: given a worksheet styled like an IQ test, can a model read the problem, infer the rule, and write the answer back into the image while leaving every other pixel untouched? It is an end-to-end test of visual reasoning through editing, where the model must both see and surgically modify only what the rule demands.
The authors built WISRD (Worksheet Image-Space Rule Discovery), a fixed 1024x1024 canvas with rendered text instructions above a frame holding the elements to operate on. There are 11 core tasks in five families: marking (intersections, midpoints, circling dots), filling, copying, counting (count dots and write the digit), and suppression ("do not edit when no edit is warranted"). On top sit eight information conditions (V0 to V7) that vary whether the in-image text, the external prompt, and reference images are present, to probe which cues the model actually uses. Four supplementary probes add multi-step spatial chains, RAVEN abstract matrices, a Wason logic task, and 4x4 Sudoku.
Scoring runs at two levels: Auto-Strict (target coverage, preservation of black geometry, text and frame, and native canvas format all required) and Auto-Loose (relaxed spatial tolerance). Each model produces 11 tasks times 50 problems times 4 no-reference conditions, 2,200 outputs in total.
The spread is wide. On the shared V0 to V3 no-reference subset, Auto-Strict pass rates are: Nano Banana Pro 48.7%, Qwen-Image-Edit 13.4%, FLUX.2 Klein 4B API 11.5%, FLUX.2 Klein 4B open-weight 11.3%, InstructPix2Pix 0.0%. Humans score 96.1% strict on the same items.
| Model | V0-V3 Auto-Strict | Auto-Loose |
| Nano Banana Pro | 48.7% | 64.0% |
| Qwen-Image-Edit | 13.4% | 20.4% |
| FLUX.2 Klein 4B (API) | 11.5% | — |
| FLUX.2 Klein 4B (open) | 11.3% | — |
| InstructPix2Pix | 0.0% | 0.0% |
| Human | 96.1% | 99.8% |
The stress probes are telling. On multi-step operation chains of 10 to 300 steps, Nano Banana Pro still holds 92%. Abstract reasoning is the real weak spot: public RAVEN matrices only 22.9%, the Wason logic task 49% exact, and 4x4 Sudoku 70%. One finding worth remembering is that models quietly read the instructions rendered inside the image. Even with no external prompt or only a generic one, scores climb when the explanation is printed in-image (under the TXT condition, Nano Banana Pro hits 88.4% on the spatial subset). An OCR-plus-Gemini-plan-then-render diagnostic pipeline scores 35% strict, beating most direct editors built from off-the-shelf parts.
For anyone building image generation and editing, WISRD is a mirror that shows whether a model actually reads the image. Editing prettily and editing to a precise rule are different things, and the latter is the precondition for treating an editor as a visual-reasoning tool. The benchmark also exposes that performance leans heavily on in-image text cues, which is both a controllability risk and a lever. For model selection, Nano Banana Pro leads this new dimension by a wide margin yet sits far below humans; spatial and abstract reasoning remain shared weak spots.
The benchmark is built from synthetic geometric problems, far from editing real photographs, so weakness here does not directly translate to failing on real images. The supplementary probes are small diagnostics by the authors' own description, so the 70% Sudoku and 22.9% RAVEN numbers warrant caution given modest sample sizes. Scoring is automatic, and although oracle and human checks are provided, the strict pass judgment is sensitive to format and geometry and may undercount answers that are right but slightly misdrawn. Only five frontier models plus one pipeline are tested, so coverage is narrow.