Unbundling Multimodal Vision Encoding
Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu
cs.CV, cs.LG
2026-11-13
From-scratch controlled training shows a vision encoder needs no pretraining and can move inside the backbone; only dedicated vision capacity matters, and encoder-free needs ~3x compute to match.
Three recipes currently compete for how multimodal language models take image input. The mainstream attaches a separately pretrained vision encoder (CLIP, SigLIP). Kimi K3 trains its encoder from scratch jointly with the language model. Encoder-free models such as Fuyu, Gemma, and Inkling drop the encoder and project raw patches straight into the backbone. The debate goes nowhere because a vision encoder bundles three distinct design choices: a visual prior from upstream pretraining, dedicated capacity reserved for vision tokens, and pre-fusion placement of that capacity before vision and text interact. Published models differ in data, compute, and architecture all at once, so cross-model comparisons isolate nothing, and prior controlled studies either compared pretrained encoders against each other or changed capacity and placement together.
This paper, from Meta FAIR and NYU, unbundles the three choices under one protocol, training every recipe from scratch.
Evaluation averages 16 Cambrian VQA benchmarks (after three epochs of instruction tuning on Cambrian-7M at 448 pixels) plus 12 language benchmarks.
| Comparison | Setting | Result |
| Scratch vs pretrained encoder | 1.7B, pretraining at 7.4–15% of budget | All four objectives tie; a 60% share loses clearly on both text and VQA |
| Free encoder pretraining | Fixed 2.7e21 multimodal FLOPs, MetaCLIP pretraining 1.4 to 9.0e21 | ImageNet linear probe 80.4% to 84.4%, while Avg. VQA is best at the shortest pretraining |
| Scratch encoder vs encoder-free | 1.7B, 2.6e21 FLOPs | 44.4 vs 39.0 Avg. VQA, gap concentrated in OCR and chart |
| Encoder-free backbone +0.4B params | wider, deeper, or both | 38.6–38.8; text improves, vision does not catch up |
| Where to put in-fusion capacity | FFN branches, first 8 layers vs alternatives | First 8 layers 45.0 (encoder 44.4); last 8 layers 35.1, every layer 39.4, spread 37.6, none beats encoder-free (39.0) |
| Scaling fits | 0.6B–6.4B ladder | Encoder-free needs 3x compute to match; in-fusion tracks the encoder at every size |
| Midtraining | 1.7B text model (1T tokens) gains vision to 1.33T | In-fusion 50.1 ≈ scratch encoder 48.6 ≈ 8-layer MoT 48.5; free MetaCLIP 43.5, encoder-free 39.8 |
| Serving latency | 1.7B, one H200, 16 concurrent requests, 448px image + 256 text tokens | Encoder-free TTFT 145 ms, in-fusion adds 16–20 ms, a separate encoder adds on the order of 100 ms |
This contradicts Shukor et al. (2025), who found encoder-free models matching or slightly beating encoders. The paper traces the disagreement to their encoder learning rate being too small, and to fitting scaling laws on an average of text, interleaved, and caption losses that dilutes the vision signal; held-out caption loss tracks VQA and shows the encoder-free deficit.
Richer patch embeddings do not rescue encoder-free either: a Gemma-style two-layer projection and an Inkling-style four-layer hierarchical MLP both fall short at 224 pixels, and the MLP matches the encoder only when compressing 640x640 images into the same 256 tokens.
Each takeaway maps to a build decision. Encoder pretraining can be dropped: joint from-scratch training loses nothing, upstream compute is wasted even when free, and attaching a pretrained encoder to a pretrained language model actually hurts (43.5 vs 48.6). The encoder-free parameter-efficiency story does not hold at matched compute; the vision deficit closes with dedicated capacity, not a wider backbone. And when time to first token matters, moving vision capacity into the first backbone layers is nearly free: VQA holds, TTFT drops sharply, and training is no slower (the encoder recipe is 3% slower at 4B).
The paper also explains industry drift: Kimi K3's from-scratch encoder route is validated, and Inkling, launched as encoder-free, went back to a 292M vision encoder in Inkling 2 Small. Methodologically, the lifetime-compute protocol with all-from-scratch training offers a template for isolating architecture variables that published-model comparisons cannot.
The authors' own list: encoder pretraining used the same images as multimodal training, so a released encoder trained on much more data could still help; only dense backbones up to 6.4B were tested, with MoE left open (authors note MoE experts could naturally host in-fusion capacity); training stayed at 224x224, leaving high-resolution and text-rich data uncovered; the data mixture was tuned for the scratch encoder at 1.7B, not per recipe; interleaved image-text and video data are unexplored.
Reasons for caution beyond that: every VQA conclusion rests on the 16 Cambrian benchmarks with Cambrian-7M instruction tuning, a single evaluation suite with no cross-suite check; absolute scores sit in the mid-40s at 1.7B, so claims at stronger baselines and longer training rely on scaling-fit extrapolation; and since all comparisons share one data pool, "pretraining does not help" reads more precisely as "pretraining on the same data does not help."