A colored-pencil benchmark finds vision models can draw, but not stop at their best frame
新智元 · wechat · 2026-07-25
TryAI ran a colored-pencil drawing arena where four vision models had to paint Mona Lisa, Starry Night, and five open-ended prompts one stroke at a time using the same tool set.
What the experiment tested
- Not text-to-image generation, but agentic drawing: the models had to plan, inspect the canvas, adjust brush/color/pressure, erase, and decide when to stop.
- Scoring used SSIM on the reference-image tasks; the open-ended prompts were judged manually.
- The setup was MIT-licensed on GitHub, so the whole benchmark can be rerun on new targets and prompts.
Main findings
- No model beat its own best intermediate draft: every copied painting ended worse than its peak checkpoint.
- More checking did not help. Gemini inspected its work the most and also suffered the biggest drop from peak score to final output.
- GPT-5.6 Sol was the most cost-efficient and overall best-rated by TryAI; Claude Fable 5 was second but much slower and about 20× more expensive.
- Grok 4.5 generated the most tokens overall, but most were cache reads, so its bill stayed low.
Cost and workflow differences
- GPT-5.6 Sol: $7.74 total, 6.2 minutes average, 29 steps.
- Claude Fable 5: $160.58 total, 12.5 minutes average, 54 steps.
- Grok 4.5: $9.21 total, 4.8 minutes average, 99 steps.
- Gemini 3.6 Flash: $12.87 total, 6.9 minutes average, 73 steps.
The broader takeaway is that these models could see their own work, but lacked a reliable “stop” decision. In a multi-step creative task, knowing when a result is already good enough became as important as planning or execution.
More from Multimodal
- A Victorian Lady prompt produces a polished AI portrait in version 8.2 — LudovicCreator · 2026-07-25
- Midjourney remixes the Tour de France into a pop-art cycling poster set — gizakdag · 2026-07-25
- Leonardo workflow uses a reference clip and Seedance 2.0 to continue videos seamlessly — techhalla · 2026-07-25
- Midjourney daily prompt series turns five recipes into a reusable style menu — tisch_eins · 2026-07-25
- ComfyUI users are debating the best way to keep characters consistent in Illustrious scenes — sparereddit1234 · 2026-07-25
- ComfyUI users want a simpler XY grid workflow for testing LoRA strength and versions — TheWebbster · 2026-07-25