A colored-pencil benchmark finds vision models can draw, but not stop at their best frame

新智元 · wechat · 2026-07-25

TryAI ran a colored-pencil drawing arena where four vision models had to paint Mona Lisa, Starry Night, and five open-ended prompts one stroke at a time using the same tool set.

What the experiment tested

Not text-to-image generation, but agentic drawing: the models had to plan, inspect the canvas, adjust brush/color/pressure, erase, and decide when to stop.
Scoring used SSIM on the reference-image tasks; the open-ended prompts were judged manually.
The setup was MIT-licensed on GitHub, so the whole benchmark can be rerun on new targets and prompts.

Main findings

No model beat its own best intermediate draft: every copied painting ended worse than its peak checkpoint.
More checking did not help. Gemini inspected its work the most and also suffered the biggest drop from peak score to final output.
GPT-5.6 Sol was the most cost-efficient and overall best-rated by TryAI; Claude Fable 5 was second but much slower and about 20× more expensive.
Grok 4.5 generated the most tokens overall, but most were cache reads, so its bill stayed low.

Cost and workflow differences

GPT-5.6 Sol: $7.74 total, 6.2 minutes average, 29 steps.
Claude Fable 5: $160.58 total, 12.5 minutes average, 54 steps.
Grok 4.5: $9.21 total, 4.8 minutes average, 99 steps.
Gemini 3.6 Flash: $12.87 total, 6.9 minutes average, 73 steps.

The broader takeaway is that these models could see their own work, but lacked a reliable “stop” decision. In a multi-step creative task, knowing when a result is already good enough became as important as planning or execution.

Original post →

A colored-pencil benchmark finds vision models can draw, but not stop at their best frame

More from Multimodal