Meta FAIR & NYU train 200+ MLLMs to show encoder pretraining is optional, vision-only params aren't
trungh98 · reddit · 2026-11-16
Meta FAIR and NYU's paper Unbundling Multimodal Vision Encoding disentangles the three choices bundled in a standard vision encoder: separate pretraining, dedicated vision-only parameters, and pre-LLM placement. They train 200+ models from scratch (LLMs included) on the same data at matched lifetime compute, mainly on Qwen3-1.7B/4B configs across 0.6B–6.4B scales.
Key findings:
- Separate pretraining is optional: encoders pretrained with four objectives at best tie a from-scratch encoder at matched compute; even free pretraining doesn't help (MetaCLIP pretrained 6.5x longer raised ImageNet linear-probe 80.4%→84.4% but not VQA).
- Vision-only parameters are required: encoder-free models trail at every size and need 3x compute to catch up; giving those parameters to the LLM improves text, not VQA.
- Pre-LLM placement is optional: "in-fusion" zero-initialized vision-only weights in the first 8 LLM layers match an encoder from 1.7B–6.4B with 13% lower time-to-first-token (45.0 VQA vs 44.4 at 1.7B).
- Findings hold when starting from a pretrained LLM via midtraining.
Bottom line: vision needs its own parameters, but not its own pretraining or encoder—addressing the current lab disagreement (Gemma 4 12B dropping the encoder, Kimi K3 training one from scratch, Inkling 2 Small bringing one back).
More from Models
- Fable 5.2 Invents 'Hard Way Day' in One-Shot Utopian AI Story — realSharonZhou · 2026-11-16
- LLMs Keep Depicting Themselves as a Floating Globe Following You Around — realSharonZhou · 2026-11-16
- Sony unveils Walrus, its most capable SpeechLLM with SOTA recognition, translation and understanding — mittu1204 · 2026-11-16
- Model Trees Launches Revamped Open-Weight Model Discovery Page — burny_tech · 2026-11-16
- Distilled image models good enough for hobbyists: 6x faster, only hands and fine text lag — Aaron_rowe81 · 2026-11-16
- Models keep getting smarter while users get babysat: a complaint about overprotective guardrails — VraserX · 2026-11-16