NVIDIA & Waterloo propose PixelUMM: encoder-free unified multimodal model on raw pixels

jiqizhixin · x · 2026-11-18

NVIDIA and the University of Waterloo introduce PixelUMM, an encoder-free unified multimodal model in an arXiv preprint. Images become spatial patches and videos spatiotemporal tubelets connected to the Transformer via single-layer projections — no pretrained vision encoder or VAE tokenizer. Separate understanding and generation FFN experts share multimodal self-attention; text is autoregressive while image/video generation uses pixel-space flow matching. Reported scores: GenEval 0.77 (0.83 with long-prompt rewriting) and 83.24 on VBench, further evidence for the de-tokenizer route to unified multimodal modeling.

Original post →

More from Multimodal

Multimodal channel →