IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation
Shaohan Yu, Jie Qin, Wende Tan, Haiwen Diao, Limeng Qiao, Yueming Lyu, Chenyang Si
cs.CV
2026-10-28
A 100K real-world interleaved text-image dataset plus a 7-metric benchmark; image-only RL lifts interleaved scores, with SenseNova-U1 SFT+RL the top open model at 73.81 mean.
Unified Multimodal Models (UMMs) can already write text and render images inside one network, but interleaved generation, where a tutorial pairs each step with a picture or a story alternates narration with illustrations, has been treated as a by-product of model unification rather than a task in its own right.
Two gaps hold the field back. On the data side, large interleaved corpora such as MMC4, OBELICS and OmniCorpus were built for multimodal understanding and carry no generation-oriented quality control; CoMM added such filtering but organizes nothing around real-world scenarios. On the evaluation side, existing benchmarks mostly hand a long interleaved response to a VLM for a single holistic score, which cannot tell you which capability is missing.
Both halves of IntReal share one starting point: a four-way scenario taxonomy keyed to how text and images cooperate, covering Planning (26.1%), Knowledge (21.2%), Tutorial (34.7%) and Story (18.0%).
IntReal-100K draws on seven naturally interleaved web and video sources: Wikivoyage, Wikitravel, Wikipedia, Xiachufang (a Chinese cooking community), and three animated series including Curious George. Curation runs from cheap to expensive:
Only about 10% of the million-scale pool survives, because one defective segment kills an entire long sequence. The yield is 100K samples, 538K images and 146.6M text tokens.
The evaluation suite selects 600 samples (150 per category) and runs two granularities. Holistic generation produces a full interleaved response from scratch. Fine-grained generation has every model produce the next image after the same real interleaved context, which makes cross-model comparison controlled. Seven metrics each get a dedicated expert: an LLM judges Text Richness and Text Logicality, an IQA model scores Image Aesthetic, DINO features drive Image Diversity and Subject Consistency, Quantity Compliance is an exact match on image count, and Context Fidelity is verified against visual rubrics that a VLM derives from the preceding context alone, before seeing the target image.
The metrics validate themselves first. Pairwise sample-level Spearman correlations stay at or below 0.42 in absolute value, agreement with human experts reaches 83.3% over 750 comparisons, and swapping the Gemma4 judge for Qwen3.6-35B-A3B leaves model-rank correlations above 0.9.
| Model | Mean | Key numbers |
| Emu3.5 | 58.95 | best open Text Logicality at 75.17; Quantity Compliance only 39.70 |
| U1-A3B-MoT | 69.43 | Context Fidelity 86.81 |
| U1-8B-MoT base | 71.88 | Quantity Compliance 95.33 |
| U1-8B-MoT+SFT | 72.31 | Text Richness +29.04%, Image Aesthetic -5.65% |
| U1-8B-MoT+SFT+RL | 73.81 | best open mean; Image Diversity +5.60% |
| NanoBanana2 (closed API) | 84.84 | +14.94% over the best open model |
Two findings carry weight. NanoBanana2 beats the best open model on Text Richness by 222.33%, yet sits below it on Image Diversity (-10.45%) and Context Fidelity (-12.13%); strong unimodal skills do not carry interleaved generation. SFT on IntReal-100K exposes the other side: every text metric improves while images degrade (aesthetics -5.65%, context fidelity -4.89%), the familiar understanding-generation conflict inside UMMs. Flow-GRPO RL on next-image generation then pulls the image dimensions back (Bagel mixes IQA, DINO and rubric rewards; U1 uses the aesthetics reward alone), adds 5.60% on Image Diversity, and lifts U1 to 73.81, the best open mean. Optimizing image generation alone improved the full interleaved task, so joint text-image training is not required at every stage.
By category, Tutorial is the hardest under fine-grained evaluation, with lower Context Fidelity and Subject Consistency; Story runs longest at about 8.7 images per sample and strains Quantity Compliance and Image Diversity. The widest open-versus-closed gap sits in the text modality, while the image gap is far smaller.
Teams building UMMs get three ready-to-use artifacts: 100K real interleaved SFT samples, a seven-dimension diagnostic protocol, and a set of metrics already validated as RL reward signals. The transfer result, that unimodal optimization improves interleaved generation, widens post-training options beyond joint SFT and gives a concrete lever against the text-image conflict. The fixed-context next-image setting also makes cross-model comparison controlled.
The authors flag two: aggressive filtering keeps only about 10% of raw data, making construction costly, and mixed post-training with unimodal data was left untested to keep the experimental setting clean.
Several caveats remain. Text Logicality is a binary judgment and the least stable across evaluators, with a 12.71-point gap for NanoBanana2 between Gemma4 and Qwen3.6. The evaluation set holds only 600 samples; the authors argue each model already forces over 6,000 generated images, but the statistical power behind seven dimensions is still limited. Only four model families are covered, with Anole and Show-o2 excluded for context length. Sources run narrow: all Tutorial data comes from one cooking site and all Story data from three low-resolution animated series, which is narrow ground for a name like IntReal. Subject Consistency is computed only for Tutorial and Story.