NVIDIA and Waterloo release PixelUMM: an encoder-free multimodal model reading and writing raw pixels
CSProfKGD · x · 2026-10-02
NVIDIA and University of Waterloo introduce PixelUMM, an encoder-free unified multimodal model for image and video understanding and generation, with code and weights released today.
- Core idea: drop both VAEs and ViT vision encoders; a single decoder-only Transformer reads and writes raw pixels directly — images as 16×16 patches, videos as 4-frame tubes.
- Architecture: a Qwen3-8B backbone with separate understanding and generation experts shares one self-attention across text, clean pixels, and noisy pixels.
- Findings: eight experiment families (F1–F8) detail ablations. F3 shows linear pixel heads produce grid-aligned patch artifacts in low-texture regions like sky at high CFG (6); released checkpoints and demos still use the default linear heads, as switching to convolutional heads requires further training.
- Project page, paper, and model are all public — a notable entry in the encoder-free unified multimodal direction.
Related event: NVIDIA Releases PixelUMM, Encoder-Free Unified Multimodal Model(3 posts)→
More from Multimodal
- Testing Qwen Image 2.1 and Minimax H3 on anime-to-photo conversion and film photo restoration — Ant_6431 · 2026-11-16
- Second take of a 'Cubee run' video generated with Minimax H3 — JahJedi · 2026-11-16
- Non-artist builds 14-chapter hand-painted storybook game with Fable 5.2 in 4 hours — jradoff · 2026-11-16
- Subject LoRAs on Minimax H3 Image learn poor likenesses while identical datasets excel on Qwen 2.1 — Ad_Organic · 2026-11-16
- Seedance-generated anime train scene stuns Reddit with cinematic quality — Ok-Solution-7889 · 2026-11-16
- AI-generated 'Men in Black' reimagined in 1977 goes viral — techhalla · 2026-11-16