TRACE-Bench decomposes multi-reference image generation into four atomic operators and finds attribute binding, not scene composition, is the bottleneck

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

Haoran Wang, Chaofan Ma, Ran Yi, Lizhuang Ma

cs.CV, cs.AI

2026-08-18

Shanghai Jiao Tong University decomposes multi-reference image generation into four atomic operators (Anchor, Disentangle, Apply, Compose) and benchmarks 9 models on 1,600 cases: even the best model scores 0.80 on attribute binding while scene composition reaches 0.91, locating the bottleneck in disentanglement and binding.

What problem this solves

Multi-reference image generation is having a moment: hand a model several reference images and a prompt, and it composes the person from image A wearing the dress from image B in the style of image C. The models are improving fast, but the evaluation tooling has not caught up. Existing benchmarks are organized around predefined task types such as subject composition or style transfer, so coverage is fragmented, case complexity is uncontrolled, and a single holistic score tells you a model is weak without telling you where.

The Shanghai Jiao Tong University team flips the perspective: stop looking at task types and look at operations instead. Any multi-reference prompt, however ornate, decomposes into a compositional formula over four atomic operators.

Method

The four operators:

• Anchor (f): locate an entity in a reference image and carry its identity-defining information into the generated image.
• Disentangle (g): extract one referenced attribute and strip it from irrelevant properties: take the dress's design, not the person wearing it.
• Apply (⊕): bind a disentangled attribute to a designated carrier, either an anchored entity from another reference or an entity described in text.
• Compose (C): arrange multiple entities into a coherent scene, optionally under relational constraints.

A prompt thus becomes a formula. "The person from image 1 wearing the dress from image 2 in the background of image 3" is C(f₁, Te⊕g₁)⊕gglobal. Structural complexity is the slot count (the number of f and g terms), and the benchmark spans slots 1 through 8.

Construction starts from roughly 50,000 candidate images filtered from Danbooru, LAION and cc12m, goes through Gemini-2.5-Pro structured tagging, balanced sampling and manual inspection to keep 3,839 images, and adds about 200 synthetic images from Nano Banana Pro for rare cases. Prompts are generated by a VLM from formula templates, filtered by GPT-5.4 (removing 4.3%) and then manually reviewed (removing another 9%). The final set: 631 formula templates, about 1,600 cases, 180 per slot level.

Evaluation is operator-aligned: each operator gets its own binary-question checklist scored by a Gemini-2.5-Pro judge: entity presence and consistency for Anchor, attribute-source correctness for Disentangle, binding correctness for Apply, coherence and leakage for Compose. Every model receives four capability scores instead of one total.

The most inventive piece is the diagnostic tree: a failed case is recursively decomposed into simpler sub-cases (drop the global reference term, split the scene into single entities, separate multi-attribute bindings), each node is re-generated and scored, and the failure is attributed to the component whose removal flipped the case to pass.

Results

Four proprietary models (GPT-Image-1.5, Nano Banana, Nano Banana Pro, Nano Banana 2) and five open-source ones (Emu3.5, FireRed Image Edit 1.1, Qwen-Image-Edit-2509/2511, OmniGen2) were evaluated:

Three findings:

• Composition is clearly the strongest capability (GPT-Image-1.5 reaches 0.9259), while Disentangle and Apply lag everywhere; even Nano Banana 2 sits at 0.7384 and 0.7989. The bottleneck is not placing content into a scene, but pulling the right attribute from the right reference and binding it to the right carrier.
• The open-source gap concentrates on g and ⊕: Emu3.5, the best open-source model, trails the proprietary leaders by roughly 0.24 on both, yet only by about 0.13 on Compose. Qwen-Image-Edit-2511 improves over 2509 on all four metrics.
• The diagnostic tree over 200 Emu3.5 cases shows joint-composition interference as the dominant localized source for entity-level failures (48.8%/43.4%/41.1% on f/g/⊕): reference items pass individually and get lost when combined. Failures on C most often localize to global-reference interference (29.0%), implicating global style constraints.

One counterintuitive detail: anchor performance varies only weakly with slot count but drops clearly as the reference image contains more entities. Reference-image clutter, not formula complexity, drives difficulty.

Why it matters

For anyone building on image generation, this benchmark provides a coordinate system that locates problems. "Multi-reference generation is unreliable" used to lead to retrying with a different model; now a failure can be traced to disentanglement or binding, and the diagnostic tree separates "cannot do it alone" from "collapses when combined"; the fixes differ. Virtual try-on and group-photo layout both fit the same formula language, so application-specific behavior can be analyzed without task-specific harnesses.

For model selection, the table is directly usable: the top three proprietary models cluster between 0.81 and 0.82 average, while the best open-source options lead their cohort but remain uncompetitive on attribute-level work.

Limitations

The paper has no dedicated limitations section; these are my own observations:

• Scoring rests entirely on Gemini-2.5-Pro as a binary VLM judge. Judge biases flow straight into the four operator scores, and no systematic human validation of the judge is reported.
• Reference images skew toward anime and illustration (Danbooru is a major source), with real-world coverage carried by a cc12m subset; sensitivity of scores to image domain is not analyzed separately.
• About 200 synthetic reference images come from Nano Banana Pro, which is itself under evaluation; potential overlap between the benchmark and the tested models' training data is not discussed.
• The authors concede that slot count measures formula structure rather than case difficulty, and the entity-count finding confirms it, yet the benchmark's controlled balancing is done by slot level only.

Terms

• Multi-reference generation: generating an image that combines content from several reference images under one text instruction.
• Attribute binding: attaching an attribute to the correct target object without leaking it onto others.
• Operator slot: the total count of anchor and disentangle terms in a formula, quantifying prompt structural complexity.
• Diagnostic tree: recursive decomposition of a failed case into simpler sub-cases to localize the failure source.
• VLM judge: a vision-language model scoring generated images against a binary question checklist.

Source

• arXiv 2608.16765v1
• PDF
• Hugging Face
• GitHub · https://github.com/Amuseum-WHR/TraceBench

Related papers

• S3D8 packs Llama 3.2 11B Vision into 2.7 bits, holding 0.661 average VQA after QAT
• InfinityEdit ignites a frozen Helios streamer with a lightweight adapter for unbounded edits
• Attention-as-classification: 1% labels match full-data PredCls on Visual Genome
• Alibaba's 6B Swift-Image Leads Open-Source Unified Editing After 243K GPU Hours
• OpticalDNA prints genomes as OCR pages and beats Enformer with 20× fewer tokens
• Self-styled Depth Anything V4 withdrawn in two days; claimed F1 +0.044 does not hold

All paper explainers