NVIDIA & Waterloo propose PixelUMM: encoder-free unified multimodal model on raw pixels
jiqizhixin · x · 2026-11-18
NVIDIA and the University of Waterloo introduce PixelUMM, an encoder-free unified multimodal model in an arXiv preprint. Images become spatial patches and videos spatiotemporal tubelets connected to the Transformer via single-layer projections — no pretrained vision encoder or VAE tokenizer. Separate understanding and generation FFN experts share multimodal self-attention; text is autoregressive while image/video generation uses pixel-space flow matching. Reported scores: GenEval 0.77 (0.83 with long-prompt rewriting) and 83.24 on VBench, further evidence for the de-tokenizer route to unified multimodal modeling.
More from Multimodal
- MAI-Image-2.7 Tops 4 of 10 Image Editing Use Cases on Artificial Analysis Leaderboards — ArtificialAnlys · 2026-11-18
- Microsoft's MAI-Image-2.7 tops image-editing leaderboard at one-sixth the price of rivals — ArtificialAnlys · 2026-11-18
- Imagination Studio turns Hermes into a local AI film studio on your own hardware — Teknium · 2026-11-18
- AI video demo studies posture, shifting gazes and small gestures — gen_ericai · 2026-11-18
- OpenAI DX engineer's article: turning GPT-6.2 Astra into a video editor with Codex — nickbaumann_ · 2026-11-18
- Blind test: four Claude models compose the same jazz-funk theme with open-source music-skill — BandicootNo3152 · 2026-11-18