Meta FAIR & NYU controlled study: visual prior unnecessary for multimodal models, dedicated capacity is key
NielsRogge · x · 2026-11-18
Meta FAIR and NYU (Saining Xie, Lucas Beyer, Luke Zettlemoyer et al.) released "Unbundling Multimodal Vision Encoding", a sharply controlled study on how multimodal models should encode images.
- Design: Published models differ in data, architecture, and compute, so they pretrain every model from scratch on the same data and compare at matched compute on VQA and language benchmarks.
- Visual prior isn't necessary: an encoder trained jointly from scratch matches a pretrained one, with no separate training stage.
- Dedicated vision capacity is what matters: encoder-free models trail on visual understanding even when the backbone is enlarged by the encoder's parameter count.
- Placement is flexible: vision-only modules added in-fusion to early backbone layers match a pre-fusion encoder and can lower time to first token.
- Findings hold across backbone sizes, even treating encoder pretraining as free.
More from Models
- Is OpenAI betting on 'good enough'? GPT-6.2 at a fraction of Opus 5.5's price — Bitter-College8786 · 2026-11-18
- Ex-OpenAI safety researchers fear system card safety chatter has gone quiet — j_asminewang · 2026-11-18
- ARC-AGI-3 maintainers explain provider adapter harnesses: vendor-recommended settings per model — mhmazur · 2026-11-18
- Opus 5.5 clones a blogger's writing style in just 30 minutes — IgorCarron · 2026-11-18
- Marin releases all 57 intermediate checkpoints of 535B-A23B MoE trained on 18.2T tokens — dlwh · 2026-11-18
- GPT-6.2 Sol voxel benchmark: mostly a regression vs 6.1, but much faster and cheaper — Angaisb_ · 2026-11-18