Kimi K3 trains its vision encoder from scratch and claims better vision evals

nrehiew_ · x · 2026-07-29

The thread says Kimi K3 is natively multimodal and that its vision encoder is trained from scratch.

Instead of training the encoder first or initializing from a contrastively pretrained model, K3 trains the whole system jointly with next-token prediction.
The author says this approach beats baselines on vision evaluations.
The attached figure contrasts Kimi K2 and K3 scaling curves and includes a plot showing the new vision tower, MoonViT-V2, with lower gradient norms and fewer spikes than the SigLIP-initialized MoonViT-3D.
The post argues that contrastive pretraining may be unnecessary as a multimodal initialization at scale.

More from Models