PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
Cong Wei, Xuanchi Ren, Bryan Chu, Weiming Ren, Huan Ling, Jiahui Huang, Laura Leal-Taixé, Sanja Fidler, Wenhu Chen, Zian Wang, Jay Zhangjie Wu
cs.CV
2026-09-30
PixelUMM drops ViT and VAE: raw patches and video tubelets enter an 8B Mixture-of-Transformers through one linear layer, covering image and video understanding and generation.
Unified multimodal models (UMMs) typically run two visual frontends: a ViT supplies semantic features for understanding while a VAE supplies reconstruction latents for generation, BAGEL being the reference design. Every conditioning image then travels as two token streams, roughly doubling visual context and the attention and memory bill, which hurts most in long-context and multi-turn settings. The dual-stream interface also mismatches the single visual stream of standard VLM pretraining, so adding generation means reworking data pipelines and training recipes.
Pixel-space modeling offered an exit: JiT and PixelDiT showed text-to-image can run on raw pixels with no latent autoencoder, and TUNA-2 and SenseNova-U1 extended that to unified image understanding and generation. Video was the missing piece. Video understanding encodes sampled frames independently; video generation leans on causal 3D VAEs that treat the first frame specially. The two conventions share no interface. PixelUMM (NVIDIA and University of Waterloo, 8B) fills that gap.
One rule governs the architecture: nothing between pixels and the backbone except linear layers.
Attention follows BAGEL's blockwise scheme: causal within text, bidirectional within each visual block, later frames seeing earlier ones; generation targets attend bidirectionally to themselves and to the prompt. Positional encoding is three-axis Native RoPE, half the heads on time and a quarter each on height and width.
The eight controlled experiment families (F1–F8) carry most of the paper's information:
| Family | Finding |
| F1 image patch size | 32×32 sees 4× more images per step at the same sequence budget, yet its training MSE stays above 16×16 |
| F2 video tubelet | Weaker spatiotemporal compression gives lower T2V loss (p32/t1 best); the final model still picks p16/t4 to match Wan2.2 VAE conventions |
| F3 decoder head | Linear heads show grid artifacts at CFG≈6; PixelShuffle is the best trade-off (5.6× FLOPs), the Wan-style decoder is cleanest but lossier and blurrier (17.1×) |
| F4 pixel vs VAE | VAE-space loss is about 4.3× the pixel loss at 10K steps, but the spaces are incomparable; gradient norms are similar and pixel training shows occasional spikes |
| F5 model size | 8B reaches the losses of 1.7B in about 1/3 the steps |
| F6 compute | 8 to 128 GPUs speeds CE about 9× in steps but MSE only 1.3× |
| F7 conditioning | With I2V and editing routed through the understanding expert, all four video benchmarks rise (+0.89 to +3.49) while image benchmarks move mixed (BLINK +2.37, SEED-I −5.41) |
| F8 interfaces | 4 FPS tubelets show no consistent advantage over 1 FPS independent frames |
Benchmark numbers (the paper states that training data differ across models, so these cannot settle architecture comparisons):
| Task | Metric | PixelUMM | Baseline |
| Image understanding | MMMU | 41.67 | Qwen2.5-VL-7B 51.30, BAGEL 55.30 |
| Image understanding | PixMo-Count | 73.03 | Qwen2.5-VL-7B 63.30 |
| Video understanding | MVBench | 70.53 | Show-o2 49.80, TUNA 54.40, Lance 62.00 |
| Text-to-image | GenEval | 0.71 with rewriter (0.55 raw) | BAGEL 0.77, also rewritten |
| Text-to-video | VBench quality | 84.10 | TUNA 84.32, Lance 85.14 |
| Text-to-video | VBench total | 68.11 | TUNA 84.06, Lance 85.11 |
Video understanding wins by the widest margin, image understanding trails dedicated VLMs, and generation lands in the usable-but-not-leading band.
For teams building unified models the value comes in two layers. The interface argument: with no ViT and no VAE, each conditioning image costs one visual stream instead of two, halving visual tokens in long-context and multi-turn settings, and the model plugs into existing VLM pretraining pipelines without adding VAE tokens. Editing and reference tasks survive through the understanding-expert routing, so fidelity no longer requires keeping a VAE around. The engineering layer: F1 through F8 hand over directly usable conclusions. Do not compress patches too hard, pick PixelShuffle decoders, expect GPU scaling to do almost nothing for generation loss (1.3×) and a lot for text (9×), and skip high-FPS tubelets for understanding for now. These come from training curves and boundary probes, not opinion.
The honest framing: benchmark results are comparable to baselines, not state of the art. The paper proves the route works on video and supplies reusable design evidence.
The authors say so themselves: understanding and generation remain two expert parameter sets sharing attention rather than fully unified parameters; p=64 patches and τ=8 tubelets are untested; cross-model comparisons are confounded by training-data differences. A few more things stand out. The released checkpoint keeps the linear output head, so the F3 convolutional-head improvements are absent from the shipped model and grid artifacts at high CFG remain. Generation specs stop at 512² and 4-second, 96-frame video, well behind dedicated generators. The VBench total of 68.11 against 81–85 for unified baselines comes from dimensions the paper does not explain. And F8 means the tubelet interface currently pays off only in pipeline simplicity, not understanding accuracy.