SolveEdit: Benchmarking Visual Problem Solving in Generative Models
Wenjie Shu, Yexin Liu, Harold Haodong Chen, Xuerui Qiu, Zehan Wang, Yidi Zhang, Yizhan Chen, Zunwei Wang, Minghao Liu, Qi Chen, Harry Yang, Xiaogang Xu
cs.CV, cs.AI
2026-09-29
SolveEdit tests 2,728 scene-transformation cases. GPT-Image-2, the strongest model, scores 57.0% SolveScore; a two-stage planner raises it to 71.6% without changing the editor.
Most image-editing benchmarks score a change the prompt already names: connect P2 to P3 with the yellow wire, and leave the rest of the image alone. Real scenes often look like "repair the broken circuit." The system has to read the image, decide what must change, render that change, and keep unrelated content intact.
Instruction-based editing measures execution of a stated transform. Newer image-to-image suites add temporal, causal, spatial, logical, and planning tasks, but they still fold three information sources into one score: the request, the current scene, and a rule drawn in the image. A single reference output adds a second problem, because several final states can be valid. SolveEdit asks whether a generator, given a source image and a goal, can recover a valid transition, realize it in the same scene, and preserve everything outside that transition. The answer is an edited image.
SolveEdit has 2,728 cases across 10 domains and 54 subdomains. The model sees only the goal and the source image. After grounding the request to visible entities, each case is labeled by the minimum extra information that fixes a valid transition:
The label is not a difficulty rating. Failures split into solution-recovery errors (an invalid transition) and visual-execution errors (right transition, wrong rendering, or protected content that moves).
Because many renderings can be valid, each case carries an atomic transition contract. Required conditions state what the output must achieve; protected conditions state what must stay intact. Each atom scores 1, 0.5, or 0, with abstention when evidence is missing. Atoms are tagged semantic (SA), relational (RA), or visual quality (VQ), then crossed with required versus protected to give six diagnostic cells. Completion R is the weighted required score; damage D is the weighted shortfall on protected atoms. The headline metric uses λ = 0.5:
SolveScore = Gquality × max(0, R − 0.5 D)
Gquality zeros missing, blank, badly corrupted, or off-request outputs. A VLM judges most atoms; a fixed subset also goes through SAM and YOLO. The generator never sees the contract.
SolveEdit-Plan is a two-stage test-time planner with no learned weights. Inspect finds unresolved variables and requests local crops as evidence. Resolve compares candidate transitions and compiles one instruction that names completion conditions and preservation duties, then calls the original generator once. Generic Vision Rewrite is the matched control: same backend, two agent calls, one final generation, unconstrained rewrite.
Eleven systems share the same manifest, one output per case. Video models are scored on the last frame. Failed generations get at most two retries; missing outputs score zero and stay in the denominator.
| Model | SolveScore | Completion R | Damage D |
| GPT-Image-2 | 57.0 | 67.6 | 23.9 |
| Seedream 5.0 Pro | 56.6 | 64.3 | 17.6 |
| Gemini 3.1 Flash Image | 55.5 | 66.0 | 23.1 |
| Qwen Image 2.0 Pro | 40.6 | 50.7 | 28.8 |
| Kling V3 (last frame) | 27.5 | 39.1 | 34.0 |
| FLUX.2 [dev] | 22.8 | 31.7 | 32.3 |
| Qwen-Image-Edit-2509 | 20.8 | 32.7 | 47.5 |
| HunyuanVideo-1.5 (last frame) | 7.4 | 17.3 | 55.5 |
Open-source image editors top out at FLUX.2 [dev] with 22.8; OmniGen2 is 9.7. Commercial systems are well ahead, and still only just over half. GPT-Image-2 has the best completion, but 6.3 points more damage than Seedream 5.0 Pro, so the two headline scores nearly tie. For all three leaders, required VQ sits above required SA and RA. A coherent picture is a weak proxy for a solved problem.
On those three models, RD trails IS by 17.2 to 23.7 points and SD trails IS by 7.0 to 8.8. After reweighting by domain mix, the IS-to-RD drop sits in required semantics (−25.0) and required relations (−13.0). Required VQ drops only 5.6. That pattern fits missing the right transition.
SolveEdit-Plan takes GPT-Image-2 from 57.0 to 71.6, 6.3 points above budget-matched Generic Vision Rewrite. Against that rewrite, completion goes from 72.8 to 75.8 and damage from 16.3 to 8.6. Mean gain across three frozen generators is 9.1 points: Qwen-Image-Edit-2509 20.8 to 26.9, HunyuanVideo-1.5 7.4 to 14.0. Planning costs two LLM calls per case, on average 3,207 input tokens and 635 output tokens. Absolute scores for the open editor and the video model stay low.
For anyone shipping an image editor or a visual agent, the split between editing and solving is the useful part. When the prompt already names the transform, commercial editors are passable. When the scene or an in-image rule has to supply the missing variable, scores fall.
The planner can sit in front of an existing editor. No fine-tuning. On GPT-Image-2 it is +14.6 over Direct and +6.3 over a call-matched rewrite, with damage cut by more than half relative to that rewrite.
This is a cleaner eval plus a modest test-time recipe, not a new generator. Open editors remain in the low twenties even with planning. The contract design, required versus protected with multiple valid finals, is the piece other visual-reasoning benches can copy.
The main paper has no standalone limitations section; the bounds sit in the protocol. Video is last-frame only, and temporal generation is out of scope. Residual errors after planning are not attributed to planning versus generation, and the open and video numbers do not show reliable task completion. Planning adds test-time compute, and two calls do not imply the same token bill as Generic Vision Rewrite. Most judging is a VLM; human agreement is not reported in the main text. λ = 0.5 is a design choice that caps damage at half of completion. Controls stop at prompt rewrite plus one generation. There is no comparison to multi-step editing or a trained planner. Editor weights never update, so execution errors pass through.