Kimi Linear: An Expressive, Efficient Attention Architecture
Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, Yulun Du
cs.CL, cs.LG
2025-10-31
Moonshot's channel-wise gated linear attention, mixed 3:1 with full attention, beats full attention on short, long, and RL tasks; KV cache down 75%, 1M-token decoding 6.3x faster.
As LLMs turn into agents and lean on RL test-time scaling, inference has to chew through long trajectories and decode a lot. Softmax attention's two old weaknesses then become hard limits: compute scales quadratically with context, and the KV cache grows linearly with length, capping throughput, context length, and real-time interaction.
Linear attention cuts the complexity but has always lost to softmax on quality. The root cause is that it stores everything in a fixed-size recurrent state (the paper calls it finite-state RNN memory), which is theoretically weak at long-sequence retrieval. Two recent ideas narrowed the gap: gating or decay, and the delta rule (DeltaNet). Pure linear structures still hit a ceiling, so the field moved to hybrids: mostly fast linear layers with a few global softmax layers to catch what linear misses. Earlier hybrids either stayed small or were not evaluated broadly, so the question of whether a hybrid can actually match or beat full attention under a fair comparison stayed open.
The core is Kimi Delta Attention (KDA). To follow it, start from DeltaNet: it reframes linear attention's state update as online gradient descent on the key-to-value mapping (the delta rule), so the state keeps correcting itself but never erases stale associations. Gated DeltaNet adds a scalar forget gate, which acts like weight decay on fast weights.
KDA swaps that scalar gate for a channel-wise one: every feature dimension gets its own forgetting rate instead of one coarse gate per head. Where GDN is a single switch for the whole room, KDA dims each lamp separately. Finer granularity means tighter control over the finite RNN memory. To compute this fine gate fast, KDA parameterizes the transition matrix as a specialized DPLR (Diagonal-Plus-Low-Rank) variant and ships a custom chunkwise-parallel algorithm that costs less than general DPLR while staying consistent with the classic delta rule.
The architecture is a layerwise hybrid: 3 KDA layers for every 1 full-attention (MLA) layer. The backbone follows Moonlight's MoE: 48B total, 3B active (8 of 256 experts plus one shared, with a dense first layer). A counterintuitive choice is that the full-attention layers run NoPE (no positional encoding) and hand all positional and recency work to KDA, which the paper argues behaves as a set of learnable position embeddings. It adds a sigmoid output gate, short conv, and head-wise RMSNorm.
All comparisons share architecture, parameter count, and training recipe (1.4T tokens), which is what lets the claims stand up. Short-context pretraining (KDA vs full MLA vs hybrid GDN-H):
| Metric | Full MLA | GDN-H | Kimi Linear |
| MMLU-Pro | 47.2 | 47.9 | 51.0 |
| MMLU | 71.6 | 72.2 | 73.8 |
| BBH | 71.6 | 70.6 | 72.9 |
After SFT, GPQA-Diamond reaches 62.1, above MLA's 57.1 and GDN-H's 58.6. At long context (128k), RULER hits 84.3 (MLA 81.3, GDN-H 80.5), with the top 8-task average of 54.5. In RL, with the same math RLVR recipe, Kimi Linear converges faster and higher on MATH500 and AIME 2025.
Efficiency is the real headline. KV cache drops by up to 75% during long-sequence generation; decoding throughput at 1M context is 6.3x higher (1.84ms per token vs MLA's 11.48ms), and prefill at 1M is 2.9x faster. The fine gate adds almost no prefill overhead, tracking GDN-H almost exactly.
This is the first hybrid linear attention to beat full attention across short-context, long-context, and RL settings under a fair comparison. For practitioners it means Kimi Linear is a genuine drop-in replacement for full attention in decoding-heavy, agentic, and long-output workloads: faster, leaner on memory, with no quality loss. Code, vLLM integration, and the 3B/48B pretrained and instruction-tuned checkpoints are open. It also backs a broader trend: linear attention with fine enough gating, plus a little global attention, can clear softmax's quality ceiling.
"Beats full attention" holds only under matched compute and recipe. The 1.4T-token comparison uses same-family, same-size baselines, not GPT-class external models. The released checkpoint is trained on 5.7T tokens and compared to Moonlight, but the paper's core conclusions rest on the 1.4T internal controls. Even at short context it does not win everywhere: EvalPlus and MATH500 sit slightly below GDN-H. LongBench V2 and Frames do not improve. The RL comparison is against MLA only, not GDN-H. KDA's fine gate and specialized DPLR add implementation complexity; the open kernel lowers but does not erase the reproduction bar relative to plain attention.