Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Penghao Wu, Haiwen Diao, Weichen Fan, Lewei Lu, Dahua Lin, Ziwei Liu
cs.CV
2026-09-02
NTU and SenseTime probe understanding-generation synergy in a native UMM. Shared compute lets one task crush the other; task-decoupled MoT lifts Geometry3K from 59.90 to 65.39.
A unified multimodal model can look at images and emit images from one network. A shared interface does not mean the two objectives help each other. They may exchange useful signal, compete for capacity, or simply coexist. Prior work disagrees: some report interference and split the pathways, others pour understanding features into generation with extra modules. Those effects are usually tangled with pretrained vision encoders, VAEs and auxiliary losses, so joint training itself is hard to isolate.
This paper uses a structurally native interface: pixels in, pixels out, no pretrained vision encoder, no image tokenizer or VAE. Qwen3-1.7B plus randomly initialized pre-buffer layers; autoregressive text, flow matching for images. Under that control, it asks when understanding and generation help, and when they fight, at representation, task and system levels.
Two routing setups first. Dense sharing sends text, clean understanding visual tokens and noised generation tokens through the same LLM layers. Modality-decoupled MoT keeps text on the pretrained branch and sends all visual tokens through a scratch visual branch. A third design, task-decoupled MoT, keeps text and clean images on the language branch and assigns noised generation images to a specialized visual branch, with shared attention and shared text semantics.
Representation-level training uses SenseNova-U1 understanding and generation data for 210k steps. Task-level cases: geometry problem solving paired with diagram generation and editing; SVG image-to-code and code-to-image; 3D spatial VQA paired with ego-motion transition, multi-view reconstruction, view-sequence completion and layout-to-image. System-level: reasoning-heavy image editing, the same examples recast as end-to-end, plan-only and execute-only, then an end-to-end UMM versus a planner-executor pipeline.
Dense sharing raises understanding and hurts generation: Vision-centric & SI 60.72 to 61.77, HPSv3 7.05 to 5.92. Modality-decoupled MoT flips the asymmetry: GenEval2 57.55 to 63.47, understanding drops across the board. Probes show generation supervision enriches visual features used for understanding, and understanding supervision raises CKA between generative features and text. Forced through one compute path, one objective dominates.
Task-decoupled MoT keeps both sides: understanding matches or beats dense joint training, GenEval2 reaches 63.96. On geometry, Und+Gen lifts Geometry3K from 59.90 to 65.39 and MathVista testmini from 71.80 to 74.30; converting the same generation data into caption-style understanding tasks helps less and less consistently. SVG joint training moves Image→SVG 80.64 to 81.70 and SVG→Image 80.21 to 86.52; blind VQA goes from 49.41 (gen-only) to 58.54. 3D spatial understanding averages 57.15 to 59.01, with better Spatial-Edit and view-sequence metrics on the generation side.
System-level, the end-to-end UMM scores 18.88 vs 16.66 overall on RISEBench and 68.33 vs 66.48 on KRIS-Bench. The logical slice of RISEBench is the exception: 9.41 end-to-end vs 11.76 for the pipeline.
Shipping one model that both understands and draws is not synergy. Synergy needs three conditions: specialize visual compute when the objectives conflict, jointly train when the tasks share domain knowledge, and go end-to-end when understanding and generation must interact inside the same decision. Default dense sharing or a hard modality split can turn one side into a regularizer. Task-decoupled routing is a working compromise, not a unique optimum; the paper flags task-routed MoE as a follow-up.
The geometry case has a sharper moral: learning to generate and edit diagrams beats learning to describe the same images in text, as auxiliary supervision for problem solving.
The backbone is 1.7B; how far this travels to larger UMMs is untested. Only the autoregressive-text plus continuous-flow recipe is covered; fully discrete or fully autoregressive variants are left open. Task- and system-level gains sit on the task-decoupled MoT, so they do not prove dense sharing can synergize. The 3D understanding lift is 1.86 points. End-to-end loses the RISEBench logical slice, so the system-level edge is not uniform. The language model is still pretrained Qwen3, not a from-scratch native multimodal train.