Four Years of MLLM Visual Encoding: From Cropping to Native Multimodal

On October 19, gabriberton posted a long thread systematically reviewing the "bitter lesson"-style evolution of vision components in multimodal large language models (MLLMs) over the past four years, and outlined directions for next-generation architectures. The core takeaway: vision encoding is moving from the traditional "resize + ViT + adapter" pipeline toward feeding entire images at native resolution—and ultimately toward native multimodal models that drop ViT entirely.

Confirmed

• Early 2023 approaches (the LLaVA era): Images were resized to medium resolution (e.g., 336×336) and passed through a ViT and an adapter into the LLM—this was the starting point of MLLM vision evolution.
• 2024–2025: multi-crop tiling becomes mainstream: Most open-source MLLMs adopted multi-crop image strategies, with representatives including LLaVA-NeXT, InternVL, Gemma 3, and Molmo, using schemes such as AnyRes, Pan&Scan, and overlapping crops.
• Rise of native-resolution approaches: The past difficulty was that ViT pretraining typically used fixed small squares of 224-448px, while native-resolution images can exceed 1k×1k; today most labs simply feed the whole image to the model at native resolution with a high token budget (e.g., 16k tokens in Qwen3-VL), with the Kimi series among other examples.
• Future direction: ViT-free native multimodality: Some models have dropped ViT altogether, with pixels entering the LLM almost directly through a single linear layer or a small MLP. Examples include Gemma 4 12B, Inkling, and Tuna-2—architectures known as native multimodal models.

Why it matters

• The thread traces the full arc of vision encoding from "bending resolution to fit ViT" to "letting models digest raw pixels directly," echoing the bitter lesson of Scaling Laws: simpler, more general architectures eventually win over clever designs as compute scales.
• For practitioners, the native multimodal route may mean components like ViT and adapters will play a diminishing role in future models—making this thread a key reference for judging multimodal architecture trends.

2026-10-19 ~ 2026-10-19 · 8 related posts

• [source] 4-year bitter-lesson evolution of MLLM vision: from LLaVA crops to native-resolution images — gabriberton · 2026-10-19
• Early MLLM recipe: LLaVA's simple 336x336 ViT-plus-adapter pipeline — gabriberton · 2026-10-19
• 2024-25 open MLLMs converged on multi-crop schemes: AnyRes, Pan&Scan and more — gabriberton · 2026-10-19
• [source] The bitter-lesson fix for MLLMs: whole image at native resolution, up to 16k tokens — gabriberton · 2026-10-19
• The multi-crop era of open MLLMs: AnyRes-style schemes with per-model hyperparameter tuning — gabriberton · 2026-10-19
• How modern MLLMs handle native resolution: heavy ViT fine-tuning plus 2D RoPE — gabriberton · 2026-10-19
• Native Multimodal Models drop the ViT: pixels flow into the LLM via one linear layer — gabriberton · 2026-10-19
• [source] Why native multimodal models won't win it all: they can't compress visual context — gabriberton · 2026-10-19