Meta-led paper: visual priors aren't necessary — dedicated vision capacity is what matters

giffmana · x · 2026-11-14

New arXiv paper "Unbundling Multimodal Vision Encoding" by Ellis Brown with Saining Xie, Luke Zettlemoyer, Lucas Beyer and others disentangles three design choices bundled inside vision encoders in multimodal LLMs: the visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity.

Because published models differ in data, architecture and compute, direct comparisons can't isolate each choice. Using a controlled framework that pretrains all models from scratch on common data and evaluates at matched lifetime compute on VQA and language benchmarks, the authors find:

The visual prior is not necessary: an encoder trained jointly from scratch matches a pretrained one, without a separate training stage.
Dedicated vision capacity is what matters: encoder-free models trail on visual understanding even when the backbone is enlarged by the encoder's parameter count.
That capacity need not precede fusion: vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and lower time to first token.

The results hold across backbone sizes, and when encoder pretraining is treated as free.

Original post →

Meta-led paper: visual priors aren't necessary — dedicated vision capacity is what matters

More from Research