NVIDIA's PixelUMM: encoder-free unified image and video understanding and generation
nvidia · hf · 2026-10-02
NVIDIA introduces PixelUMM, an encoder-free unified multimodal model that performs image and video understanding and generation directly in pixel space.
- Motivation: existing UMMs use separate visual representations for understanding vs. generation, inflating visual context length and complicating integration with vision-language pretraining; extending pixel-space modeling from images to video is non-trivial due to differing temporal representations
- Method: images as spatial patches, videos as spatiotemporal tubelets, connected via single-layer linear projections to a shared backbone; a Mixture-of-Transformers architecture combines shared attention with task-specific parameters, extending clean-pixel prediction to video generation and jointly supporting autoregressive text prediction and pixel-space flow matching
- Results: competitive performance across image and video understanding and generation, plus ablations on decoder design and spatiotemporal patch size
Related event: NVIDIA Releases PixelUMM, Encoder-Free Unified Multimodal Model(3 posts)→
More from Multimodal
- Testing Qwen Image 2.1 and Minimax H3 on anime-to-photo conversion and film photo restoration — Ant_6431 · 2026-11-16
- Second take of a 'Cubee run' video generated with Minimax H3 — JahJedi · 2026-11-16
- Non-artist builds 14-chapter hand-painted storybook game with Fable 5.2 in 4 hours — jradoff · 2026-11-16
- Subject LoRAs on Minimax H3 Image learn poor likenesses while identical datasets excel on Qwen 2.1 — Ad_Organic · 2026-11-16
- Seedance-generated anime train scene stuns Reddit with cinematic quality — Ok-Solution-7889 · 2026-11-16
- AI-generated 'Men in Black' reimagined in 1977 goes viral — techhalla · 2026-11-16