Kimi K3’s vision path uses MoonViT-V2 and 4× token downsampling
stochasticchasm · x · 2026-07-28
- The image shows Kimi K3’s vision pipeline: visual inputs are first encoded by MoonViT-V2 and then projected into the LLM with a lightweight MLP projector.
- MoonViT-V2 is described as a 27-layer vision transformer with about 0.4B parameters, using RMSNorm and removing bias terms to stabilize training.
- For images and videos, the model shares parameters and factorizes attention into intra-frame spatial and inter-frame temporal passes, with temporal pooling compressing tokens further.
- A 2×2 pixel-shuffle downsampling step cuts the number of visual tokens by 4×, making inputs up to 3584×3584 pixels feasible within a 1M-token context.
More from Research
- CMU’s O-VAD detects industrial video anomalies by tracking object state over time — CarnegieMellonU · 2026-07-28
- Kimi K3 technical report shows 2.5× better scaling efficiency than Kimi K2 — stochasticchasm · 2026-07-28
- Latent Action Model talk will show how to learn Super Mario from observation alone — ceciletamura · 2026-07-28
- GLM 5 follows up with per-head Muon to balance attention-head updates — stochasticchasm · 2026-07-28
- MLA-based KV cache costs 12 GB per million tokens, with KDA state at 230 MB BF16 — zephyr_z9 · 2026-07-28
- OpenAI chart says 43.5% of occupation-specific ChatGPT use goes beyond the user’s job — soumitrashukla9 · 2026-07-28