HiUni: Harmonizing Understanding and Generation in Shared Visual Space
Jiahao Hu, Jingdong Du, Yichen Zhang, Yi Song, Zhenyu Zhang, Jiaxin Niu, Ruihang Li, Xiangzhao Hao, Yiqian Zhang, Yong Huang, Zefeng Zhang, Zheyuan Gu, Shuohuan Wang
cs.CV, cs.AI
2026-11-19
HiUni turns Qwen3-VL-4B into a unified model with one shared ViT and backbone plus a 0.9B diffusion head; understanding stays at 75.56 (vs 75.61) and Chinese generation leads UMMs.
Unified multimodal models (UMMs) aim to handle image understanding and image generation in one model. In practice almost everyone splits the architecture first: Janus assigns a separate visual encoder to each task (a language-aligned ViT for understanding, a reconstruction-oriented VAE for generation), and Bagel goes further with a Mixture-of-Transformers that routes task-specific computation inside the backbone. The split exists because the two objectives conflict, but it also means the two visual spaces never talk to each other. Unification stays at the parameter level, not the representation level.
This paper from Baidu's ERNIE team asks whether the split is necessary at all: can a pretrained VLM be extended with generation, with no separate encoders and no task-specific backbone computation?
HiUni initializes from Qwen3-VL-4B-Instruct. Understanding and generation share the same ViT, the same post-merger visual tokens, and the same dense Transformer backbone; task-specific computation lives entirely on the output side, with the original LM head for text and a new Pixel Diffusion Head for images. Total size is 4B + 0.9B.
Two problems had to be solved, and the two core modules map onto them.
First, pixel-space diffusion is hard to optimize. HiUni skips VAE latents and diffuses directly on pixels, and in early experiments a standard DiT with block-wise AdaLN produced frequent loss spikes. The Pixel Diffusion Head therefore removes AdaLN from all blocks, reducing to a plain stack of Transformer blocks; timestep conditioning enters through a time token processed by the shared backbone, with a single modulation kept before the final prediction. Each visual token predicts a 32×32×3 pixel patch.
Second, the pretrained ViT abstracts away spatial detail, and the patch merger doubles the effective patch size from 16 to 32. HiGate is a lightweight residual path that takes raw pixels of the noisy image before patch embedding, projects them to the diffusion head's hidden dimension, and injects them token-wise through a learned gate. Semantics stay in the backbone; spatial detail arrives at the last moment.
Training runs in three stages: 30K warmup steps with all pretrained components frozen, 310K joint pretraining steps with generation resolution climbing from 256² to 2048² and a text:understanding:generation ratio of 0.1:0.3:0.6, then 22K SFT steps that add image editing. Generation uses x-prediction with a velocity objective and resolution-dependent noise scales.
Understanding: 75.56 average over seven benchmarks, against 75.61 officially reported for the Qwen3-VL-4B base, essentially no regression; OmniDocBench stays at 0.24.
| Comparison | Metric | Result |
| Qwen-Image-Bench Chinese Overall | generation | 47.63, highest among compared UMMs; English 50.52 |
| OneIG English Overall | generation | 0.538 vs 0.561 for the strongest UMM |
| ImgEdit Overall | editing | 3.66, second among unified models (UniSpace 3.82) |
| GEdit CN/EN Overall | editing | 6.70 / 6.81, second (SenseNova-U1 at 7.05 / 6.94) |
| LongCat-Image-Edit | editing specialist | 4.02, with Qwen-Image-Edit-2511 at 4.01, still clearly ahead |
The most informative experiment is the controlled comparison in Section 4.3, using models trained with far fewer steps than the release model. Dense-2B vs MoT-2B, same recipe and initialization, differing only in whether the backbone is shared: understanding 58.55 vs 59.23, OneIG 0.291 vs 0.345. Task routing does help generation at equal scale. But spending a similar total budget on a larger shared dense backbone (Dense-4B, also 4B+0.5B) wins all six understanding benchmarks (65.04 vs 59.23) while keeping generation competitive at 0.314. Single-task baselines quantify the cost of joint training: generation-only reaches 0.390 on OneIG vs 0.314 jointly, while understanding is slightly higher under joint training (65.04 vs 64.58).
The HiGate ablation is clean: removing the residual path drops OneIG to 0.267, ViT features give 0.284, raw pixels 0.314, with understanding flat near 65 across all three settings. Gate strength grows toward the clean-image end of denoising, matching the intuition that fine detail matters late. Adding editing supervision lifts understanding from 62.64 to 64.49 while OneIG dips from 0.538 to 0.528, so positive transfer and interference coexist.
This cuts the path of extending a mature VLM into a unified model to its minimum: no split encoders, no duplicated backbone, no MoT, and under 1B of new compute, all at the output side. Teams building UMMs can inherit the full pretrained assets and infrastructure of an existing VLM. The controlled study also yields a budget rule: at similar total parameters, scale the shared dense backbone rather than splitting into task branches. You win understanding across the board and lose only a little generation. An incremental improvement, but the evidence is solid and the recipe is directly usable.
Generation remains the weak side. It loses to MoT under matched settings (0.291 vs 0.345), and joint training costs 0.076 OneIG against the generation-only baseline; the shared backbone shrinks interference without eliminating it. Editing trails specialist models by more than 0.36 points.
The understanding comparison is not fully clean: 75.56 is set against an official Instruct number and a re-evaluated thinking variant, and the drops on MMBench and OCRBench are attributed to training data distribution without further evidence. Most baselines are official numbers from their own papers, judged by GPT-5.2, so cross-model comparability is limited.
The AdaLN removal rests on two development-run loss curves that the authors themselves flag as qualitative only, collected at different resolution schedules and stages. Training compute is not disclosed, so the cost of 310K joint steps at up to 2048² resolution can only be guessed. The authors also concede the paper stays at the capability-coexistence level; mutual gains between understanding and generation are still early signals.