YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction
Shengyu Wang, Tengyue Nong, Junming Liu, Haosong Peng, Siyu Li, Ruiqi Lei, Fushuo Huo, Yuhao Pan, Yunfeng Fan, Ding Wang, Nan Cheng, Haozhao Wang, Wenchao Xu
cs.CV
2026-11-04
YANAMI has 402,639 multi-turn dialogues that require cross-turn state. Fine-tuning BAGEL raises the benchmark from 65.02 to 78.25, with editing up 10.3% and GenEval up 2%.
Unified multimodal models, UMMs, put image understanding, generation, and editing in one set of weights. Public data for them is still mostly single-turn. In recent sequential-editing sets and interleaved image-text chats, many turns collapse once the past images are pasted into the prompt: the turn rewrites as an ordinary one-shot edit. Nothing there forces the model to remember a ban on new accessories, or a requirement the user just took back.
People do not edit that way. They change their minds, add constraints, and point at an earlier frame. The assistant has to keep a dialogue state that is still valid, then render it. YANAMI, Your Assistant Needs to Assimilate Multiturn Interaction, is training data and a benchmark for that bookkeeping. The project is led by HKUST.
Images come from four public sets: Zebra-CoT, WEAVE, MICO-150k, and MACRO. GPT-5.6 Sol, Qwen3.7-Plus, and Gemini 3.1 Pro run a coarse filter that rejects only severe corruption or unreadable content. Repairable images go to GPT-Image-2 or Nano Banana 2 and through the filter again. Surviving images are pooled by scenario and written into multi-turn dialogues with scenario and capability prompts. An automatic pass checks logic and image-text agreement. Failures are edited or regenerated.
State moves differently in each scenario. Multi-Image Fusion composites several references from the history. Image Editing continues from earlier changes and keeps them by default. World Modeling changes viewpoint or action while geometry and object relations still have to hold. Visual Reasoning applies a hypothesis step by step and answers count and constraint questions.
Six skills are written into the dialogues: coreference, history retrieval, cross-turn consistency, spatial and physical consistency, constraint following, and updates after a user correction. Each of the 402,639 training dialogues uses at least four. The bar is explicit state management: a target turn cannot be rewritten as a single-turn task by supplying the historical images. On the paper's comparison table, WEAVE and ChatUMM miss that bar.
YANAMI-Bench has 400 dialogues and 1,883 images, 100 dialogues per scenario. SHA-256 hashes show no exact image duplicate against the training set. Three annotators with graduate STEM training check every sample. Dialogue State Understanding, DSU, is four questions per dialogue, on reference, retrieval, constraints that are still active, and updates after corrections: 1,600 questions. Visual Quality is a perceptual score from 0 to 10. State Consistency splits the still-valid state into atomic checks, 8,467 across the benchmark, and a check scores only when it is visibly met.
Models that both understand and generate are weighted 4:4:2 on DSU, state consistency, and visual quality. Models that only draw are weighted 6:4 on the last two. GT-History replaces earlier generated images with ground truth. Self-History puts the model's own images back into context.
The best open UMM is Emu3.5 (34B) at 67.41. The best open editor is FLUX.2 at 67.30. Closed systems score 83.05 (GPT-Image-2) and 80.93 (Nano Banana 2). Visual Reasoning is the weakest scenario for most models. Among open UMMs, Show-o2 sits at 39.29.
BAGEL-7B moves from 65.02 to 78.25 after fine-tuning on YANAMI-400K, a gain of 13.23 points. The benchmark shares no exact image with the training set, so the gain is not recall of the same picture. The four scenarios go from 62.44, 67.40, 68.16, and 62.06 to 78.98, 84.96, 73.26, and 75.80. Once reasoning reaches 75.80, world modeling at 73.26 is the low cell. Average DSU rises from 81.44 to 95.69, and editing-scenario understanding rises from 76.50 to 98.00.
| Method | Metric | Result |
| BAGEL → +YANAMI | YANAMI-Bench overall | 65.02 → 78.25 |
| BAGEL → +YANAMI | DSU average | 81.44 → 95.69 |
| BAGEL → +YANAMI | GenEval overall | 0.73 → 0.75 |
| BAGEL → +YANAMI | GEdit-EN-full | +10.28% |
| Emu3.5 / FLUX.2 / GPT-Image-2 | Bench overall | 67.41 / 67.30 / 83.05 |
POPE moves from 86.5 to 87.4 and MMBench from 85.0 to 89.1, while MME-P falls from 1647.9 to 1616.9. On GenEval, counting goes from 0.70 to 0.73, position from 0.47 to 0.52, and attribute binding from 0.49 to 0.55. Single-object and color each drop by 0.01. The paper records the GenEval overall change, 0.73 to 0.75, as a 2% gain.
A right answer about state still drops details in the picture. All nine UMMs fully answer 78 constraint questions and 71 correction questions. After requiring an aligned generation check for every model, the shared subsets are 77 and 66 dialogues. Realization on that slice stays clearly short of perfect. GT-History already blocks error carried forward from earlier pictures, and the last generation turn still trails the first by 2%, 6%, and 6% for BAGEL, FLUX.2, and InternVL-U. Under Self-History, BAGEL and Qwen-Edit sit another 5.16% and 3.77% below their GT-History scores.
The judge is Qwen-3.8-Flash-Next. Four annotators scored 100 samples each from fine-tuned BAGEL, Emu3.5, and Nano Banana 2. Pearson correlation with humans stays above 0.80, and agreement with GPT-5.6 Sol is high as well.
78.25 is still below GPT-Image-2 at 83.05. Closed models are scored 6:4 on images alone. UMMs are scored 4:4:2, with understanding included. The two totals are not a ranking.
Multi-turn editing, multi-image fusion, and viewpoint changes stall when a constraint has to survive later turns. YANAMI-400K holds 2,047,236 images and 2,643,291 turns, averaging 6.56 turns and 5.08 images per dialogue, with a maximum of 43 turns. The training mix is 36% fusion, 25% editing, 28% world modeling, and 11% visual reasoning. One hundred benchmark dialogues per scenario make the comparison direct.
A team that already has a UMM can fine-tune on this set and leave the architecture as it is. The reported BAGEL-7B-MoT run freezes the VAE and trains the language model, the vision encoder, and the understanding module, at learning rate 1e-5, cross-entropy weight 0.25, generation MSE weight 1.0, and a 16,384-token cap. Editing moves more than generation, in line with how much of the data is continued editing under old constraints.
Ground-truth history with perfect past images will overstate stability on a real rollout.
Frontier models wrote the dialogues and produced much of the imagery. GPT-Image-2 and Nano Banana 2 are the illustrators inside the pipeline and the closed references on the benchmark. How much of the gain is imitation of those two editing styles is not separated out. Humans checked all 400 benchmark dialogues. The roughly 400K training dialogues passed automatic checks, so quiet state errors can enter the gradient. SHA-256 removes exact image copies and does not catch near-duplicates.
Self-History drops are reported only for BAGEL and Qwen-Edit. The main text defines the understanding-realization gap on a shared subset. Per-model gap numbers sit in an appendix table, and the main figure stays qualitative.
Four GPT-Image-2 requests were rejected by a content policy, so its score uses 948 valid outputs. Nano Banana 2 uses 939. Missing records are not filled in as zeros. Closed averages count only the outputs that came back.
Visual reasoning is 11% of training, and it is where most baselines score lowest. Fine-tuning lifts that cell from 62.06 to 75.80, with thinner supervision than fusion. MME-P drops 31 points. The understanding scores that rise are POPE and MMBench.