Kimi K2.5 Scaling Methodology

AI寒武纪 · wechat · 2026-07-18

This article recaps Moonshot AI CEO Yang Zhilin's GTC speech on the "scaling methodology" behind Kimi K2.5. The main theme divides model capability scaling into three dimensions: token efficiency, long context, and agent quantity.

Token efficiency: The team detailed the Muon optimizer and its distributed implementation, claiming it can replace Adam and yield better training results under identical token/parameter constraints. They also used QKclip to resolve attention logit explosion and loss divergence during large-scale training.
Long context: They proposed KimiLinear / KimiDeltaAttention, mixing linear and full attention, and changed the original scalar decay to a diagonal matrix, balancing performance and efficiency across short-context, long-input, and long-output tasks.
Agent swarm: Designed an orchestrator + sub-agent training paradigm, paired with instantiation rewards, completion rewards, and outcome rewards, aiming to enable multiple agents to collaborate on complex tasks in parallel.

The article also highlighted key aspects of K2.5:

Trained on over 15 trillion tokens on an H800 cluster;
The first open-source model trained jointly on native vision + text, emphasizing that "early fusion" is superior to text-before-vision;
Visual and textual capabilities mutually enhance each other, achieving strong performance even with almost no visual SFT;
The speech concluded with a teaser for the next-gen architecture direction: analogizing "residual connections" to a temporal recurrent structure, and further evolving it into "attention residuals."

Original post →

Kimi K2.5 Scaling Methodology

More from Models