Kimi K3 trains MoonViT-V2 from scratch to stabilize multimodal training

nrehiew_ · x · 2026-07-29

The thread highlights a key change in Kimi K3: the vision encoder, MoonViT-V2, is trained entirely from scratch with next-token prediction.

Earlier practice, including Kimi K2.5, initialized the vision encoder from a contrastive model such as SigLIP.
The authors say that once a pre-trained encoder is attached to the LLM, joint optimization becomes unstable.
MoonViT-3D with SigLIP initialization shows persistent gradient spikes, while MoonViT-V2 stays more stable during training.
The claim is that training with next-token prediction lets the encoder be shaped directly by the language-modeling objective.
The post says MoonViT-V2 matches the SigLIP-initialized baseline on vision evaluations, suggesting contrastive pretraining may not be necessary at scale.

More from Models