How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Lin Chen, Bolin Ni, Qi Yang, Lan Jiang, Kun Ding, Xiaoran Fan, Hower Yang, Ying Wang, Shiming Xiang
cs.CV
2026-09-28
On a shared 11-rung MoE ladder, encoder-free MLLMs trail encoder-based ones on multimodal loss but improve faster; fitted scaling laws put the crossover near 6×10²¹ FLOPs.
Nearly every production MLLM (Qwen3-VL, Gemma 3, LLaVA) bolts a pretrained ViT such as SigLIP in front of the language model, so the visual prior comes from large-scale image-text pretraining rather than the run at hand. A smaller line of work deletes the encoder and feeds projected raw patches straight into the decoder: Fuyu, EVE and SOLO proved feasibility, and Gemma 4 12B Unified and Inkling now ship the design at product scale. The architecture is simpler and free of the encoder's fixed-resolution, fixed-token-budget constraints. What nobody had measured is how much training compute that simplification costs, and whether the gap widens or closes with scale. This paper, from CASIA, UCAS and Tencent's foundation model team, runs the controlled experiment.
The setup's strength is that everything is shared except the front end: an 11-rung sparse MoE decoder ladder (1.1B-44B total, 71M-2.4B active parameters; 256 routed experts, top-8, plus one shared expert), the same 1:1 text/multimodal data mixture, the same optimization, and one visual token per 32×32 pixels on both sides. The encoder-based variant uses a 400M SigLIP 2 ViT with a ConvPool adapter and causal attention throughout. The encoder-free variant uses a patch projection adapted from Gemma 4 12B Unified, with bidirectional attention among visual tokens within each image and causal attention elsewhere. Both consume the same pixels and pass the same number of visual tokens, so the measured gap is attributable to the encoder itself.
Scaling laws follow Chinchilla's IsoFLOP recipe: fix the budget, vary model size, fit validation loss as a quadratic in log size, take the vertex. Six budgets per objective (2×10¹⁹-2×10²⁰ FLOPs for text, 10²⁰-10²¹ for multimodal) give the allocation law Mopt ∝ C^a and the frontier L = E + KC^−γ, with a shared entropy floor E per objective. Equal-loss comparisons use two ratios, EGC (compute) and EGM (model size), where below 1.0 means the encoder-free side is worse. Overtraining, meaning a fixed model trained on k× the optimal tokens, is modeled as a prefactor shift with an empirically fitted multiplier.
Text is the clean control: the two loss frontiers nearly overlap (γ 0.0973 vs 0.0979), EGC sits around 0.98 and EGM around 0.99. Deleting the encoder barely touches pure-text efficiency. On the multimodal objective the gap is real but narrowing:
| Metric | Encoder-based | Encoder-free |
| Allocation exponent a (multimodal) | 0.464 | 0.570 |
| Loss-compute exponent γ (multimodal) | 0.2998 | 0.3778 |
| EGM at equal loss | 1 | ≈0.80 (≈1.25× FLOPs per token) |
| Predicted crossover | — | 6.1×10²¹ FLOPs (80% interval 4.2×10²¹-1.0×10²²) |
For scale, Kimi K2.5 pretraining is about 10²⁵ FLOPs, so the crossover sits roughly three orders of magnitude below flagship budgets. Under 5× overtraining the crossover moves to 1.2×10²² FLOPs and EGC drops from 0.62 to 0.52: encoder-free models prefer spending compute on a larger decoder, so extra tokens at fixed size benefit them less. The shift toward larger models also survives switching to causal attention over visual tokens (a = 0.557), so it is not an artifact of the attention mask. By topic the gap varies widely: STEM, mostly language and symbols, is near parity at the largest fitted budget (EGC 0.71→0.95), Charts follows (0.51→0.69), while perception-heavy Caption (0.18→0.34), GUI (0.34→0.50) and OCR (0.28→0.57) lag far behind.
Three probes show the decoder absorbing the encoder's job. During training, the encoder-free multimodal loss falls slowly and then drops sharply; across that drop, attention to visual tokens at layer 12 of the 8B model rises from 0.217 to 0.645, near encoder-based levels. Loss is computed only on text tokens, so uninformative visual tokens get ignored early and learn slowly until they attract enough attention, at which point learning accelerates. Layerwise, encoder-free visual tokens diverge from their inputs in shallow layers, so the early decoder acts as an implicit visual encoding stage, while text trajectories match across architectures. Expert routing for visual tokens grows more concentrated (higher MaxVio across the four largest models), consistent with a subset of experts taking over vision. An ablation also finds the multimodal benefit of bidirectional attention grows with compute, matching the contextualization role the ViT used to play.
On 11 downstream benchmarks (3-shot), encoder-free models still trail at every measured scale: at 100B tokens the 33B model averages 51.2 vs 57.7, with the gap narrowing as compute grows, consistent with the loss-side findings.
This is the first quantitative basis for deciding whether to delete the visual encoder. Below roughly 10²¹ FLOPs the encoder prior pays off on every topic, so small and mid-budget teams have no reason to switch. For frontier-scale pretraining, the fitted laws say the encoder's advantage runs out near 10²² FLOPs, well inside practical budgets, and the payoff is a single unified transformer with no fixed-resolution bottleneck. The decoder probes also carry a design implication: current decoders inherit language-first designs, and the growing benefit of bidirectional visual attention, shallow visual processing and expert specialization suggests decoders built for native vision could pull the crossover earlier. The honest framing: this is a scaling-law extrapolation, not an observed catch-up at that scale.
Stated by the authors: the multimodal fit covers 10²⁰-10²¹ FLOPs, so the crossover extrapolates about 6× beyond the data, under three assumptions, namely that the power law persists, the ViT stays fixed in size, and the entropy floor depends only on the data distribution. Their sensitivity analysis concedes the extrapolated crossover moves noticeably with the assumed irreducible loss, though the exponent ordering (encoder-free falling faster) survives every perturbation, and leave-one-budget-out refits keep the crossover between 4.3×10²¹ and 9.8×10²¹ FLOPs.
Points that stand out on a careful read. The ViT is fixed at 400M while decoders grow, so its share of capacity shrinks along the ladder; every trend is conditional on that regime, and jointly scaling the encoder is a different, unstudied allocation problem. The 1:1 data mixture is held fixed, so mixture effects on encoder-free scaling are untested. Validation loss is the primary metric, and encoder-based models win every downstream benchmark at measured scales; there is no benchmark evidence at the crossover. Held-out checks at 2.5-5× extrapolation landed within about 2% error, which is reassuring but not the same as a 6× jump.