Kimi K2.5 Scaling Methodology

AI寒武纪 · wechat · 2026-07-18

This article recaps Moonshot AI CEO Yang Zhilin's GTC speech on the "scaling methodology" behind Kimi K2.5. The main theme divides model capability scaling into three dimensions: token efficiency, long context, and agent quantity.

• Token efficiency: The team detailed the Muon optimizer and its distributed implementation, claiming it can replace Adam and yield better training results under identical token/parameter constraints. They also used QKclip to resolve attention logit explosion and loss divergence during large-scale training.
• Long context: They proposed KimiLinear / KimiDeltaAttention, mixing linear and full attention, and changed the original scalar decay to a diagonal matrix, balancing performance and efficiency across short-context, long-input, and long-output tasks.
• Agent swarm: Designed an orchestrator + sub-agent training paradigm, paired with instantiation rewards, completion rewards, and outcome rewards, aiming to enable multiple agents to collaborate on complex tasks in parallel.

The article also highlighted key aspects of K2.5:

• Trained on over 15 trillion tokens on an H800 cluster;
• The first open-source model trained jointly on native vision + text, emphasizing that "early fusion" is superior to text-before-vision;
• Visual and textual capabilities mutually enhance each other, achieving strong performance even with almost no visual SFT;
• The speech concluded with a teaser for the next-gen architecture direction: analogizing "residual connections" to a temporal recurrent structure, and further evolving it into "attention residuals."

Related event: Yang Zhilin Shares Kimi K2.5 Scaling and Efficiency Strategies(4 posts)→

Original post →

More from Models

• PostTrainBench creator notes Muse Spark 1.3 system card files it under loss-of-control risk — maksym_andr · 2026-10-27
• Mathematicians debate AI math writing: abysmal today, improving shockingly fast — miniapeur · 2026-10-27
• LessWrong: GLM models show outlier chain-of-thought controllability vs Kimi and DeepSeek — tomekkorbak · 2026-10-27
• AI deference trap: agent spots medical red flag but stays silent to avoid questioning the clinician — echen · 2026-10-27
• The trust paradox: only the least trustworthy LLMs will do the sketchiest things — AaronBergman18 · 2026-10-27
• Claude's refusal to type one-time passwords is a real security impediment — AaronBergman18 · 2026-10-27

Models channel →