Can Visual Generation Expand What Unified Multimodal Models Know?
Wenkai Li, Wenyi Hong, Zhuoyi Yang, Jie Tang
cs.CV, cs.AI
2026-10-15
Facts taught to unified multimodal models only via image generation can be drawn but seldom answered in text; interleaved image-text alignment lifts GLM-Omni QA accuracy to 26.9%.
Unified multimodal models (UMMs) are sold on the idea that generation and understanding reinforce each other. The evidence so far is mostly benchmark-level: Janus-Pro and BAGEL post strong scores on separate understanding and generation evals, which says nothing about whether a specific fact learned while generating an image later becomes usable in language. Prior work on modal aphasia showed the two can come apart, with models reproducing a visual concept while failing to describe it in words. This paper measures that seam directly: inject knowledge only through image-generation training, then ask whether text-only QA can retrieve it.
The hard part is control. Pretraining corpora contain almost everything, so without isolation you credit the model with knowledge it already had.
Knowledge Transfer Probing (KTP) turns knowledge into injectable, testable units:
The three models span distinct architectures: BAGEL uses a Mixture-of-Transformer-Experts design with separate branches, SenseNova-U1 models pixels directly with no separate visual encoder or VAE, and GLM-Omni starts from the GLM-4.1V-Thinking base with a discrete image tokenizer.
For the alignment experiments, entities are split in half into an alignment set and a test set. The alignment set gets either 10 text-only QA examples per entity (Textual Alignment) or full responses that generate an image, inspect it, and answer (Interleaved Image-Text Alignment). The test set keeps only T2I supervision and is always evaluated with text-only QA, so gains there measure generalization to entities whose answers were never taught.
| Model | Direct (text) | Visual | Textual align | Interleaved |
| BAGEL | 8.86 | 31.43 | 15.43 | 17.14 |
| GLM-Omni | 14.57 | 54.00 | 12.57 | 26.86 |
| SenseNova-U1 | 20.00 | 61.14 | 28.00 | 28.57 |
| Random guessing | 15.96 |
Direct transfer is weak. Only SenseNova-U1 beats random guessing in most categories; BAGEL and GLM-Omni sit at or below it. Visual accessibility runs far higher: SenseNova-U1 hits 88% on number with 0% in text. What the model can draw, it usually cannot say.
Interleaved alignment lifts the average for all three models, GLM-Omni most of all (12.57 to 26.86, with shape 12 to 64 and spatial 0 to 40). Textual alignment alone actually drops GLM-Omni from 14.57 to 12.57. An ablation that masks the image loss inside interleaved sequences falls from 26.86 to 13.71, so the generation loss itself, not the answer format, carries the transfer.
At inference, Visual CoT (generate an image, then reason over it) raises GLM-Omni to 41.14 on average, and the number category from 0% to 4%, 36%, and 20% across the three models. Conditional analysis shows correct generation predicts correct answering, but even after interleaved alignment the accuracy given a correct image stays below 50% for every model.
This is unit-level controlled evidence for a claim usually argued from benchmark deltas: image-generation training can install knowledge that text supervision never supplied. The practical levers are concrete. Supervision organization matters a lot, with interleaved single-response data clearly ahead of separate QA examples. Visual CoT is a decoding-time change that costs no training. And the near-zero transfer on number and text, categories where the knowledge is text embedded in images, is a flag for anyone designing data mixes: this stuff does not cross modalities on its own.
The authors list three: fictional synthetic data only, with no natural images; architectural effects not isolated, since the three models also differ in training steps and data ratios; alignment data and inference strategies left open. More issues show up on close reading. All seven categories are low-level visual attributes, with no semantic or factual knowledge, so extrapolation is risky. Visual accessibility relies on a VLM judge, adding its own noise. Interleaved alignment is not uniformly positive (SenseNova-U1 color falls from 20.0 to 4.0), so averages hide per-category regressions. And on number and text, test entities stay near 0% while directly supervised alignment entities answer at 72% to 96%: taught answers come out, visually learned ones do not.