4-year bitter-lesson evolution of MLLM vision: from LLaVA crops to native-resolution images
gabriberton · x · 2026-10-19
A long thread by gabriberton traces the 4-year bitter-lesson evolution of the visual side of multimodal LLMs:
- 2023 (LLaVA era): images resized to medium resolution (e.g. 336×336), passed through a ViT and adapter into the LLM.
- 2024–2025: multi-cropping became the norm — LLaVA-NeXT, InternVL, Gemma 3, Molmo all used variants (AnyRes, Pan&Scan, overlapping crops), adding complexity and per-model hyperparameter tuning.
- Today's trend: most labs now just feed the whole image at native resolution up to a token cap (e.g. 16k tokens in Qwen3-VL); examples include Qwen3-VL, Kimi-VL, Gemma 4.
- The future: a Meta paper argues for encoder-free Native Multimodal Models (NMMs) where pixels enter the LLM via a linear layer (Gemma 4 12B, Inkling, Tuna-2). The author disagrees, predicting more encoders instead, noting NMMs can't compress visual data and thus handle long contexts inefficiently.
Related event: Four Years of MLLM Visual Encoding: From Cropping to Native Multimodal(8 posts)→
More from Models
- Claude Opus 5.5 in agent mode gets multiple testing corrections backwards, user finds — Sauers_ · 2026-10-19
- User Slams Claude's Slack Integration as Worst-in-class for Personal Accounts — akumar · 2026-10-19
- DeepSeek V4.1 Flash matches Astra on 3D, but vision quality still lags rivals — bookwormengr · 2026-10-19
- Investor urges Google to distill Argon into a small model priced at 50% of Luna to top OpenRouter — bindureddy · 2026-10-19
- DeepSeek V4.1 Flash becomes first open model to crack top 10 on search leaderboard — gaganghotra_ · 2026-10-19
- KV Cache Compression Makes 1M Context Manageable at Under 100GB Per User — khademinori · 2026-10-19