UMM-JEPA: predicting continuous perceptual representations beats discrete codes for unified multimodal models
Yucheng Zhou · hf · 2026-11-17
UMM-JEPA (University of Macau & Tencent Youtu Lab) reformulates generation in unified multimodal models as progressive masked next-scale prediction of continuous visual-understanding representations, with pixel rendering decoupled and attached afterward.
- Conventional objectives (discrete codes, latent flows) optimize synthesis-oriented spaces mismatched with perceptual understanding, so generative training fails to improve understanding.
- Controlled comparisons show the representation-prediction objective accounts for most of the gains; a data-matched control leaves means nearly unchanged.
- UMM-JEPA hits a 16-benchmark visual mean of 51.71 vs 46.36 for the understanding-only Qwen-VL baseline, with text scores also rising and stronger transfer than discrete-code or latent-flow objectives.
- Linear probes and visual-evidence interventions reveal richer object/spatial structure and better use of image-specific information.
More from Multimodal
- Ink Studio launches free on GitHub for 3D pixel-art Gaussian splat animation — ring_hyacinth · 2026-11-17
- PKU & Tencent Hunyuan's MovieRoll extends streaming video generation to 60s with 78% KV-cache savings — Yunyang Ge · 2026-11-17
- GlyphAtlas disentangles stroke detail from layout for accurate scene text synthesis — Luozheng Qin · 2026-11-17
- 'veo-experimental' spotted in hidden docs, suggesting a new Google Veo model is in the works — koltregaskes · 2026-11-17
- Built with ChatGPT: Forge extension tames 30,000-LoRA collections with incremental loading — Artistic-Account-423 · 2026-11-17
- AI-generated Godzilla-style kaiju video of Tokyo street panic impresses — Careful-Loquat-8117 · 2026-11-17