ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation
Jiahao Zhao, Xiaomin Yu, Zhongxiang Sun, Fengwei Teng, Chengwei Qin, Xiaobin Hu, Jun Xu, Shuicheng Yan
cs.CV
2026-08-05
Open-world image generation needs search, reasoning, then drawing. ToolArtist turns both into autonomous actions of one unified multimodal model, trained by RAD-GRPO with complementary intent and quality rewards.
Text-to-image models look good but fail on open-world tasks that need complex semantics, multi-hop reasoning, long-tail knowledge, or time-sensitive information; the critical facts are not in the prompt and cannot be stored in static parameters. Prior work that adds agents to image generation either hard-codes a search-then-synthesize workflow or lets the agent only rewrite the prompt and offloads actual generation to an external generator. In both, generating the image is not an action the model can autonomously choose. ToolArtist argues that genuinely agentic generation requires reasoning, tool use, and image generation to be autonomous actions of one policy.
The backbone is Emu3.5, a natively autoregressive unified multimodal model (UMM) that predicts text and images as next tokens. Two tools: text search (Google plus an LLM reader with cross-page fallback) and image search (filters undownloadable pages, judges relevance with an LLM reader, returns reference images with source-aware summaries).
Stage one, SFT. A teacher agent equipped with search tools and an external image generator (gemini-3-pro-image-preview) produces multi-turn rollouts (reasoning, tool call, observation, alternating, not a fixed search-then-draw). A converter rewrites each trajectory into UMM format: the external generation tool call is concealed, but the generated image is retained as native visual tokens, so it looks as if the model drew it. Trajectories that fail on image loading, tokenization, or context length are filtered, leaving 7,132 training trajectories. The trick is making the model believe it can draw, rather than knowing it called an external generator.
Stage two, RAD-GRPO (Reason-Act-Draw GRPO). The full policy is optimized online over real inference trajectories, with reward a weighted sum of intent and quality rewards (default 0.5 each). The intent reward judges whether the final visual caption is accurate and executable enough that an ideal generator would produce the right image. The quality reward is a four-dimension world-knowledge reward model: faithfulness 0.1, visual correctness 0.4, text accuracy 0.4, aesthetics 0.1, with text accuracy falling back to 0.5 when no readable text is needed. Format reward, draw signal, length penalty, and a no-draw penalty prevent collapse to search-only. The key architectural point is that the generated image stays in the multimodal history, so drawing does not end the interaction; the model can inspect, re-search, and redraw.
Two benchmarks: WISE (1,000 prompts, 25 subdomains, combining consistency, realism, aesthetics) and WorldGenBench-Humanities (732 prompts, KCS measures knowledge-checklist satisfaction).
| Metric | ToolArtist | Comparison |
| WISE overall | 0.79 | BAGEL+CoT 0.70, Emu3.5 0.65 |
| WGB-H avg KCS | 22.10 | Unify-Agent 15.58, Qwen-Image 21.76 |
ToolArtist is the strongest non-proprietary model in the open-source set. Frontier proprietary models still lead: Nano Banana reaches 0.89 on WISE. Its strength is knowledge-heavy natural-science categories (physics 0.81, chemistry 0.79); its weakness is reasoning-heavy spatio-temporal ones (time 0.62, space 0.75). A striking ablation: removing the source-aware image-search summaries drops WISE from 0.79 to 0.61, with biology alone falling 0.50, showing heavy dependence on search quality. The SFT corpus has 7,132 trajectories, averaging 20.5k tokens and 4.0 tool calls.
For agentic multimodal work, folding reasoning, retrieval, and native generation into one policy and training it with a reusable RL infrastructure is a useful paradigm, with SFT data and training infra released. The SFT trick of concealing the tool while keeping the image is a general way to fold any agent trajectory into a UMM.
There is no dedicated limitations section, and several points deserve skepticism. Evaluation covers only two benchmarks, a narrow surface; spatio-temporal reasoning (time 0.62, space 0.75) is clearly behind the proprietary frontier (0.89), strong on retrieval-heavy tasks and weak on reasoning-heavy ones. It depends heavily on online search quality, with summaries both a key contribution and a key dependency. No training compute, latency, or GPU hours are reported, so claims of being cheap cannot be checked. The RL reward comes from live online search trajectories, which the authors admit fluctuate noticeably.