Meta FAIR paper: multimodal LLMs don't need pretrained vision encoders, just dedicated vision capacity

meta · hf · 2026-11-14

Meta FAIR and NYU's paper "Unbundling Multimodal Vision Encoding" decomposes vision encoders into three bundled design choices — a visual prior (upstream pretraining), dedicated vision capacity, and pre-fusion placement — then ablates each in a controlled from-scratch pretraining setup at matched lifetime compute:

The visual prior is not necessary: an encoder trained jointly from scratch matches a pretrained one without a separate stage.
What matters is dedicated vision capacity: an encoder-free model trails on visual understanding only when the backbone lacks it, and adding capacity matching the encoder's parameter count closes the gap.
Capacity need not precede fusion: vision-only modules added in-fusion to early backbone layers perform comparably and can lower time-to-first-token.
Findings hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained LM.

Original post →

Meta FAIR paper: multimodal LLMs don't need pretrained vision encoders, just dedicated vision capacity

More from Multimodal