Native Multimodal Models drop the ViT: pixels flow into the LLM via one linear layer

gabriberton · x · 2026-10-19

Some models now drop the ViT entirely: pixels go almost directly into the LLM through a single linear layer or a small MLP, as in Gemma 4 12B, Inkling and Tuna-2.

These are called Native Multimodal Models (NMMs), going a step beyond the current approach of heavily fine-tuning a ViT with 2D RoPE inside the MLLM.

Related event: Four Years of MLLM Visual Encoding: From Cropping to Native Multimodal(8 posts)→

Original post →

More from Multimodal

Multimodal channel →