NTU Paper Maps When Understanding and Generation Actually Synergize in Unified Multimodal Models

机器之心 · wechat · 2026-09-14

A new paper from MMLab at NTU Singapore examines whether visual understanding and generation genuinely help each other in Unified Multimodal Models, across three levels.

Representation: DenseSharing boosts understanding but hurts generation; a decoupled MoT reverses the effect. Each objective provides useful learning signals for the other, but forcing a shared compute path makes one dominant. The proposed Task-decoupled MoT routes understanding tokens to a language-anchored branch and generation tokens to a specialized branch, linked via text semantics and shared attention.
Task: Mutual transfer happens when tasks share underlying knowledge. Case studies on geometric reasoning, SVG (Image→SVG 80.64→81.70, SVG→Image 80.21→86.52) and 3D spatial intelligence all show bidirectional gains that also generalize to MathVerse/MathVista. Converting generation data to captioning yields no such benefit—learning to generate is itself distinct supervision.
System: On reasoning-intensive image editing, end-to-end UMM beats a modular Planner→Executor pipeline on RISEBench and KRIS-Bench, suggesting end-to-end optimization is itself a source of capability.

The authors argue UMMs matter most for tasks sharing latent knowledge (e.g., embodied AI as a VLA+WAM base model) and agentic generation scenarios where understanding and generation must interact continuously.

Original post →

NTU Paper Maps When Understanding and Generation Actually Synergize in Unified Multimodal Models

More from Multimodal