NVIDIA and Waterloo release PixelUMM: an encoder-free multimodal model reading and writing raw pixels

CSProfKGD · x · 2026-10-02

NVIDIA and University of Waterloo introduce PixelUMM, an encoder-free unified multimodal model for image and video understanding and generation, with code and weights released today.

Core idea: drop both VAEs and ViT vision encoders; a single decoder-only Transformer reads and writes raw pixels directly — images as 16×16 patches, videos as 4-frame tubes.
Architecture: a Qwen3-8B backbone with separate understanding and generation experts shares one self-attention across text, clean pixels, and noisy pixels.
Findings: eight experiment families (F1–F8) detail ablations. F3 shows linear pixel heads produce grid-aligned patch artifacts in low-texture regions like sky at high CFG (6); released checkpoints and demos still use the default linear heads, as switching to convolutional heads requires further training.
Project page, paper, and model are all public — a notable entry in the encoder-free unified multimodal direction.

Original post →

NVIDIA and Waterloo release PixelUMM: an encoder-free multimodal model reading and writing raw pixels

More from Multimodal