NVIDIA's 15B PixelUMM drops VAE and vision encoder, learns image understanding and generation from raw pixels
lmoroney · x · 2026-10-06
NVIDIA researchers released PixelUMM, a 15B-parameter unified model that handles image and video understanding and generation directly from raw pixels — no VAE for generation and no pretrained vision encoder for understanding.
Key points:
- Images are cut into 16x16 pixel patches and fed into a decoder-only Transformer built on Qwen3-8B
- Generation uses iterative denoising, bypassing the standard latent-diffusion VAE design
- Code is Apache 2.0, but the checkpoint is licensed for non-commercial research and evaluation only
The author suggests reading it alongside latent diffusion to reconsider what each component actually buys, and testing small-text rendering and memory use on your own GPU before forming an opinion.
Related event: NVIDIA Open-Sources PixelUMM, a 15B Encoder-Free Unified Multimodal Model(2 posts)→
More from Multimodal
- One prompt, four approvals: Claude produces a 78-second cartoon explainer video — alex_verem · 2026-10-25
- Canva's new Craft image model plans layouts first and splits images into editable layers — alex_verem · 2026-10-25
- Local RTX 5090 test: FLUX.1 Dev keyframes + Wan 2.1 14B I2V nail neck folds and wet-scale physics — LatentForge-Dev · 2026-10-25
- Credit: the Qwen3.5-4B 3D generation model is by community dev ccore2 — victormustar · 2026-10-25
- Lure-TTS: 600M voice models trained for under $200, running 4x faster than real time — smirkishere · 2026-10-25
- MCP turns AI podcast production into a workflow — the hard part is listener retention — aziz4ai · 2026-10-25