Meta & NYU paper: vision prior unnecessary, dedicated capacity placed early is what matters
burny_tech · x · 2026-11-16
A Meta/NYU paper (Unbundling Multimodal Vision Encoding, Ellis Brown, Hu Xu, Luke Zettlemoyer, Saining Xie et al.) disentangles three bundled design choices of vision encoders in a controlled, matched-compute framework:
- The visual prior is unnecessary: an encoder trained jointly from scratch matches a separately pretrained one, skipping the extra stage.
- Dedicated vision capacity is what matters: encoder-free models trail on visual understanding even when the backbone is enlarged by the encoder's parameter count; across 0.6B–6.4B models, encoder-free setups need 3x more training compute to match.
- Capacity need not precede fusion: vision-only modules added in-fusion to early backbone layers match a pre-fusion encoder and cut time-to-first-token by up to 13% in a 0.4B-vision-capacity setup.
The findings favor dedicated vision capacity trained jointly with the backbone and placed early inside it.
Related event: Meta and NYU: pretrained vision encoders not required for multimodal LLMs(2 posts)→
More from Multimodal
- 1.8GB open TTS model Tontaube V1.1 clones voices and narrates audiobooks on CPU alone — solyarisoftware · 2026-11-16
- Fable 5.2 renders photorealistic 3D scene: 1M tokens, 5hrs render — Dr_Singularity · 2026-11-16
- Fable 5.2 turns a Starship launch into an entirely JavaScript painting — brandon_galang · 2026-11-16
- Sony unveils Walrus, its most capable SpeechLLM with SOTA recognition, translation and understanding — mittu1204 · 2026-11-16
- Fable 5.2 tries mosaic art style, lands a bridge with 14 unique spans — repligate · 2026-11-16
- An endless cinematic loop: Arabian warrior and a jinn made with Minimax H3 and ACE-Step — magratheya_64 · 2026-11-16