Reasoning with Image Generation
Nishad Singhi, Hector Garcia Rodriguez, Aditya Arora, Marcus Rohrbach, Anna Rohrbach
COLM 2026
cs.CV
2026-09-15
TU Darmstadt's ReImaGin lets MLLMs call an image generator mid-reasoning. It beats text-only CoT and Visual Sketchpad on six tasks, by up to 25 points on path tracing.
When multimodal models answer visual questions, the actual reasoning usually happens in text. The image is described, then a textual chain-of-thought takes over. That is a poor fit for questions that need a visual operation: how many pieces sit behind a black square, how two overlapping room photos sit in one floorplan, which object a vehicle hits if it drives forward.
Tool-augmented agents try to fix this with a fixed catalog of vision modules (crop, depth, detection, segmentation). Visual Sketchpad is the main baseline of that design. Each module does one pre-specified transform. Generating a novel view, deleting an occluder, or redrawing dashed lines as solid ones is not in the catalog. The recipes for calling those tools are also hand-written in-context examples, redone whenever the task changes.
ReImaGin from TU Darmstadt and hessian.AI wraps an instruction-following image generator as one free-form tool, generateimage. The agent runs a ReAct loop: write a thought, then emit Python. That code can call the generator or cheap helpers such as crop, overlay, subtract, and numpy. The generator takes a natural-language command plus optional reference images and returns a new picture into context. The run ends when the model writes TERMINATE.
The same tool does different work by prompt. Depth items get a depth map. Occlusion counting inpaint-removes the black square. Collision prediction draws a line along the object's heading. Multi-view spatial questions stitch two photos, then ask for a top-down map. Path tracing turns dashed connectors solid. Puzzle completion strips whitespace and subtracts pixels to isolate the missing piece.
Image generation is noisy, and hard transforms vary more. On MMSI, each call samples 10 candidates and Gemini-3.1-Pro picks the one that best matches the prompt. The other five tasks barely change across samples, so selection is off.
A second thread asks if the visual strategy can be searched rather than written. Headline numbers still use human in-context recipes. Automatic discovery starts from a prompt that has tools but no task policy. A proposer reads success and failure traces, offers 5 candidates per round for up to 4 rounds, with at most 50 train and 50 dev items. The reasoner and generator stay frozen. One-time cost is about $214 per task.
Three MLLM backbones: Gemini-3.1-Pro, GPT-5, and open-weights Qwen-3.5-27B. The default generator is Nano-Banana-Pro (Gemini-3-Pro-Image). Baselines are a no-tools direct answer and Visual Sketchpad's specialist toolbox. All scores are accuracy except occlusion counting, which uses sMAPE (lower is better), averaged over three seeds.
GPT-5 shows the widest gap:
| Task | No tools | Sketchpad | ReImaGin |
| Path tracing | 48.5 | 41.7 | 73.5 |
| Puzzle (MIRA) | 29.5 | 16.7 | 44.9 |
| Occlusion sMAPE↓ | 12.5 | 11.7 | 7.4 |
| Spatial (MMSI) | 36.0 | 37.7 | 43.7 |
| Collision | 44.7 | 51.0 | 54.0 |
| Depth (BLINK) | 83.9 | 88.7 | 89.2 |
Path tracing is +25 points over no-tools, the ceiling quoted in the abstract. Occlusion sMAPE drops by about 40% relative. On puzzle and tracing, Sketchpad falls below no-tools: those tasks need generative edits the specialist stack cannot do.
With Gemini, tracing rises from 71.5 to 89.0, but depth loses to Sketchpad's dedicated depth estimator (94.6 vs 99.2). Qwen tracing jumps from 35.7 to 78.5. Turning on image-side selection for MMSI lifts Gemini from 51.0 to 59.0 and Qwen from 42.0 to 54.3. Baselines get 20-sample majority vote for matched compute; ReImaGin still edges them.
Replacing generateimage with a no-op that returns the input unchanged, keeping the rest of the scaffold, drops all six tasks (tracing 89.0 to 77.5, puzzle 42.3 to 36.5). The gain is the generated picture.
Open generators FLUX.2 and Qwen-Image-Edit are fine for local edits: occlusion sMAPE 7.0 and 6.8, next to Nano-Banana-Pro's 7.1. FLUX.2 depth falls to 89.1, under the no-tools 91.7. Qwen-Image-Edit tracing is 76.0 against 89.0.
Automatic strategies beat the no-strategy tool setup on all five Gemini tasks. Tracing 87.3 vs handcrafted 89.0, depth 94.3 vs 94.6. The searched recipes often look like the human ones: a depth map for BLINK, an arrow from the vehicle nose for collisions. The composite MMSI recipe (stitch, then top-down) does not come back; the search finds a panorama-like stitch instead. Adding color to the tracing strategy slightly hurts because the generator fails the coloring more often than it fails making dashes solid.
A lot of visual reasoning needs an intermediate picture of a counterfactual view, not another detector. The stack is training-free and modular, and every step is an inspectable image. Occlusion completion, trajectory extrapolation, multi-view fusion, and making messy lines readable all share one natural-language interface.
The bill is explicit. MLLM reasoning costs $0.04 per instance for both ReImaGin and Sketchpad. ReImaGin adds $0.12 of image generation and median latency goes from 52 s to 76 s. Searching a strategy is about $214 per task once. The extra spend buys tool flexibility from a frontier image model.
On depth, a specialist estimator is still more accurate. The added coverage is one generator in place of a hand-built tool per transform, plus a path to search the strategy instead of writing every example.
A hand audit of 120 intermediate images found 75% faithful to the request. Final accuracy is 87% when the image is faithful and 43% when it is not. Failures split in two. The generator misplaces a sink, door, and stairs on a floorplan, or draws a heading line perpendicular to a motorcycle. Or the image is right and the MLLM still misreads it, picking a small purple bus beside the object the trajectory actually hits.
Specialist depth remains more accurate. With Qwen as the reasoner, open generators fall to or below no-tools on collision and spatial reasoning; global geometry is still the weak spot.
Evaluation sets are small. Puzzle test has 26 items. MMSI uses 50 items per split. Collision and occlusion are 100-sample draws. Path tracing is a synthetic task designed by the authors, dashed lines that pass close without crossing, which inflates the value of making the lines solid. Headline results still depend on human strategies. Automatic search never recovers the composite MMSI policy. A Gemini-discovered occlusion prompt transferred unchanged to Qwen gets worse.