UniSpace: Unified Visual Representation and Scalable Multimodal Modeling
Jinbo Yan, Limeng Qiao, Jie Qin, Junyan He, Feize Wu, Guanglu Wan
cs.CV, cs.AI
2026-08-09
A reconstruction patch path on a frozen ViT cuts PR-DINOv2 rFID from 0.57 to 0.14; 8B UniSpace then does generation and editing in one visual space, no VAE.
Semantic vision encoders such as CLIP and SigLIP are the default visual interface for multimodal understanding, and generation systems reuse them as semantic conditioners. Final-layer tokens are optimized for abstraction, so pixel detail is hard to recover. Production stacks therefore keep two visual spaces: a semantic ViT for understanding and a VAE latent for generation. Architecturally unified models such as BAGEL still inherit that split inside a Mixture-of-Transformer-Experts backbone.
The Meituan group asks whether a pretrained semantic ViT can host one visual space for understanding, generation, and editing. Their claim is specific: frozen Transformer blocks can carry fine detail. The original patch embedding is what suppresses it.
A diagnostic run replaces SigLIP2's pretrained patch embedding with a random linear map, freezes every Transformer weight, and keeps the reconstruction probes identical. Last-layer PSNR rises from 20.96 to 24.66, a 3.70 dB gain. At the patch-embedding output both maps still reconstruct, about 39.3 versus 39.7 dB; the split appears only as tokens travel through the same frozen blocks. Semantic classification climbs from 0.93% to 81.29% with depth, while pixel recoverability on the pretrained path collapses. The random map disrupts that semantic trajectory and leaves more low-level variation in the residual path. Randomizing the patch layer is not a product: it destroys semantics. It does show that a different input parameterization can open another route through the same frozen weights.
Patch Reparameterization adds that route on purpose. The original embedding Ps and the Transformer blocks stay frozen, preserving pretrained semantic tokens Ts. A reconstruction-aware embedding Pr, initialized from Ps, shares the same frozen blocks and yields Tr. Tr is projected to 128 dimensions and concatenated with Ts along the channel axis, producing the unified representation Tu. PR-SigLIP2 and PR-DINOv2 are 768+128=896-d; PR-Qwen-ViT is 1152+128=1280-d.
An MLP merge is the thing they refuse. An entangled variant reconstructs encoded latents well (rFID 0.069, PSNR 33.83, zero-shot 78.53 versus SigLIP 79.10). After a DiT is trained on that merged space, the high-fidelity reconstruction decoder yields FID 120.9, while the semantic decoder yields 8.07. About 95% of merger variance is explained by the semantic path, so the generative prior never learns the directions the reconstruction decoder needs. Explicit concatenation makes those directions addressable under flow matching.
Reconstruction training updates only Pr, the compression layer, and a ViT-XL decoder. The semantic path is untouched. Generation uses component-balanced flow matching with reconstruction weight λr=0.75. UniSpace then attaches PR-Qwen-ViT to a Qwen3-8B Mixture-of-Transformer-Experts model: an understanding expert and a generation expert, hard-routed, with shared self-attention at every layer. Reference images, targets, and samples all live in the same frozen tokenizer. There is no second VAE. Training walks 256, 512, then 1024 resolution plus SFT, about 510 million sample instances and 470 billion multimodal tokens, using roughly 142K NPU-hours on Ascend 910B.
On ImageNet-1K 256 validation, PR-DINOv2 records rFID 0.14, PSNR 30.84, SSIM 0.90. Against RAE on the same DINOv2-B backbone, rFID falls from 0.57 to 0.14 (75.4% relative), PSNR rises from 18.86 to 30.84, and SSIM from 0.48 to 0.90. PR-SigLIP2 moves rFID from 0.53 to 0.18. Both beat RAEv2's DINOv3-L K=7 trade-off setting (PSNR 22.57, rFID 0.29), and PR-DINOv2's rFID is also below FLUX-VAE at 0.18 and SD-VAE 3 at 0.20.
Under LLaVA-v1.5 with Vicuna-7B and a frozen encoder, the projector sees the full Tu. PR-SigLIP2 averages 64.37 versus 63.39 for SigLIP2-B; PR-Qwen-ViT averages 68.94 versus 68.29. Semantics survive the extra reconstruction stream.
Generation is the other side of the trade-off. PR-DINOv2 with an 839M DiT reaches gFID 2.10 without CFG and 1.87 with CFG 1.2; matched RAE is 1.51 and 1.13. Better reconstruction makes a harder latent for the prior.
At system scale, 8B UniSpace scores 4.28 on ImgEdit overall, against 3.20 for 7B BAGEL and 4.51 for 20B Qwen-Image-Edit-2511. Hybrid editing is the weak cell at 2.70. GEdit is 7.41 English and 7.38 Chinese, bilingual 7.39, above BAGEL at 6.51 and close to SenseNova-U1 8B at 7.47; semantic consistency is strong, perceptual quality weaker. GenEval overall is 0.84 versus 0.82 for BAGEL and 0.91 for SenseNova-U1, with counting at 0.69 and position at 0.83. DPG-Bench is 86.49, with the best relation score in the table at 94.97. OneIG-Bench is 0.561 English and 0.533 Chinese, bilingual 0.547, and the style scores are the highest among listed unified models.
Unified multimodal models often share a backbone while still keeping two visual tokenizers. This paper moves the change into the patch embedding and a channel-wise concat, so a frozen semantic ViT can keep both meaning and detail, and an 8B system can drop a separate VAE. Anyone who wants understanding and generation in one stack, and already has a SigLIP or Qwen-ViT, gets a cheaper path than retraining a vision backbone.
It does not beat RAE on generation FID, and it does not beat a specialized 20B editor on ImgEdit. The offer is one frozen visual space that brings all three jobs to a usable level.
The authors say UniSpace is tuned for generation and editing. System-level understanding still trails dedicated VLMs, and they do not treat UniSpace VLM scores as primary evidence. Because the encoder also has to support generation, it cannot be freely unfrozen on understanding data. Hybrid editing at 2.70 and GenEval counting at 0.69 show that compositional edits and counting remain shaky. On GEdit, perceptual quality lags semantic consistency. On the tokenizer side, better reconstruction comes with worse gFID, so the unified latent is not free for the generative prior. Training data is internally curated, and 142K NPU-hours is not a weekend reproduction.