DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Donghao Li, Dongjie Ji, Erhang Li, Fang Wei, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong Dai, Guangbo Hao, Guanting Chen, Guoai Cao, Guolai Meng, Guowei Li, Han Yu, Han Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoling Zhang, Haoming Luo, Haoran Wei, Haotian Yuan, Haowei Zhang, Haowen Luo, Haoyu Chen, Haozhe Ji, Hengqing Zhang, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, J Yang, JQ Zhu, Jia Luo, Jia Song, Jia Yu, Jialiang Huang, Jialu Cai, Jian Liang, Jiangting Zhou, Jiasheng Ye, Jiashi Li, Jiaxin Xu, Jiewen Hu, Jieyu Yang, Jin Chen, Jin Yan, Jingchang Chen, Jingli Zhou, Jingting Xiang, Jingyang Yuan, Jingyuan Cheng, Jingzi Zhou, Jinhua Zhu, Jiping Yu, Joseph Sun, Jun Ran, Junguang Jiang, Junjie Qiu, Junlong Li, Junmin Zheng, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexing Zhou, Kezhao Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Leyi Xia, Li Zhang, Liang Zhao, Lihua Guo, Lingxiao Luo, Linwang Ma, Linyan Zhu, Litong Wang, Liyu Cai, Liyue Zhang, Longhao Chen, MS Di, MY Xu, Max Mei, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Mingxu Zhou, Minmin Han, Ning Wang, Panpan Huang, Panpan Wang, Peixin Cong, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Qiwei Jiang, Rui Tian, Ruifan Xu, Ruijie Lu, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runqian Chen, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, Ruyi Chen, SH Liu, Shanghao Lu, Shangmian Sun, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoqing Wu, Shaoyuan Chen, Shengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Shuying Yu, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye, Tianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tun Wang, W Zhang, WL Xiao, Wangding Zeng, Wei An, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun Gao, Wenkai Yang, Wenlve Huang, Wenqing Hou, Wentao Zhang, Wenting Ma, Xi Gao, Xiang He, Xiangwen Wang, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xinyu Yang, Xinyu Zhang, Xu Chen, Xuanyu Wang, Xuecheng Su, Xueyin Chen, Xuheng Lin, Xuwei Fu, YC Yan, YQ Wang, YW Ma, Yanfeng Luo, Yang Zhang, Yanhong Xu, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Xu, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Qian, Yi Shao, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yijia Wu, Yiliang Xiong, Yiling Ma, Ying He, Ying Tang, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiang Zhang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo, Yongtong Wu, Yu Wu, YuKun Li, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuanhao Li, Yuduan Wang, Yuehan Yang, Yuer Xu, Yuhan Wu, Yuhao Meng, Yuheng Zou, Yukun Zha, Yunfan Xiong, Yupeng Chen, Yuping Lin, Yuqian Cao, Yuqian Wang, Yushun Zhang, Yuting Yan, Yutong Lin, Yuxian Gu, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuxuan Zhou, Yuyang Zhou, Yuzhen Huang, ZF Wu, Zehao Wang, Zehua Zhao, Zehui Ren, Zekai Zhang, Zhangli Sha, Zhe Fu, Zhe Ju, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zheren Gao, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhixian Huang, Zhixuan Chen, Zhiyu Wu, Zhizhou Ren, Zhongyu Wu, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihua Qu, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang, Ziwei Xie, Ziyi Gao, Ziyi Wan, Zizheng Pan, Zongqing Yao
cs.CL, cs.AI
2026-04-26
DeepSeek发V4预览版(Pro 1.6T/49B、Flash 284B/13B,均原生支持1M上下文),靠混合稀疏注意力CSA加HCA、流形约束超连接mHC和Muon优化器,把百万token的单token推理FLOPs压到V3.2的27%、KV缓存到10%。V4-Pro-Max在知识、推理上刷新开源SOTA。
推理模型带来的 test-time scaling 把上下文越拉越长,而原始注意力的计算量随序列长度的平方涨,百万 token 成了难以承受的瓶颈。DeepSeek 这篇放的是 V4 系列预览版,目标就一个:在不牺牲能力的前提下,把超长上下文的效率壁垒打破,让百万 token 上下文成为日常可用的东西。
V4 保留了 V3 的 DeepSeekMoE 和多 token 预测(MTP),主要改三处。
万亿参数 MoE 训练不稳是个硬骨头,他们靠两招压住:Anticipatory Routing 把路由网络和主干网络解耦(用历史参数算路由索引、提前缓存,且只在 loss spike 时动态启用,整体开销可忽略);SwiGLU Clamping 把 SwiGLU 线性项夹在 [-10, 10]、gate 上限 10,消掉离群值。后训练阶段把原来混合 RL 整个换成在线蒸馏(OPD)来合并各领域专家,并用生成式奖励模型(GRM,actor 自己当裁判)替代标量奖励模型。
两个模型,V4-Pro 1.6T 总参/49B 激活、V4-Flash 284B/13B,分别在 33T、32T token 上预训练,都原生支持 1M 上下文。
效率(1M token 设定,对 V3.2):
| 单 token FLOPs | KV 缓存 | |
| V4-Pro | 27% | 10% |
| V4-Flash | 10% | 7% |
base 模型对比里,V4-Flash(13B 激活)用更小参数在多数 benchmark 上超过 V3.2-Base(37B 激活),尤其世界知识和长上下文;V4-Pro-Base 进一步拉开。
后训练的 V4-Pro-Max:知识上 Simple QA 57.9(比所有开源高约 20 个点,但落后 Gemini-3.1-Pro 的 75.6)、Chinese-Simple QA 84.4;推理上 LiveCodeBench 93.5、Codeforces rating 3206(超 Gemini 3052、GPT-5.4 3168,人类选手里排第 23),作者称这是开源模型首次在编程竞赛上追平闭源;长上下文 MRCR 1M 83.5 超过 Gemini-3.1-Pro、仅落后 Opus 4.6。agent 任务和 K2.6、GLM-5.1 打平、略输闭源。Putnam-2025 形式化数学跑到 120/120。
DeepSeek 是开源里少有的把「百万 token 常态化」真做出来的,而且是在更省算力和显存的前提下,这对长程 agent、跨文档分析、test-time scaling 都是基础设施级利好。V4-Pro-Max 在知识和推理上把开源 SOTA 往前推了一档,编程竞赛追平闭源是个明确的进展。对做长上下文部署、推理基础设施的人,这套 CSA 加 HCA 的混合注意力值得认真看。
作者承认架构相对复杂(为降风险保留了不少已验证组件和 trick,未来要再精简);Anticipatory Routing 和 SwiGLU Clamping 有效但底层原理还没搞明白。定位上他们自己说,推理上仍落后最强闭源约 3 到 6 个月,agent 任务也还差闭源一截。这是预览版,不少东西还在迭代,多模态、长程多轮 agent 都是未来工作。注意 benchmark 里不少是自家内部评测框架跑的,跨模型严格一致性作者有声明但外部难以复核;FP4 乘 FP8 在现有硬件上峰值算力和 FP8 乘 FP8 一样,所谓的进一步提速要等未来硬件。