Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, Wenfeng Liang, Ying He, Yuqing Wang, Yuxuan Liu, Y. X. Wei
cs.DC, cs.AI, cs.AR
2025-05-14
Trained on 2,048 bandwidth-capped H800s, DeepSeek-V3 uses MLA to cut KV cache to 70 KB/token and MTP to raise generation speed 1.8x.
HBM capacity grows less than 50% a year. Model memory demand grows more than tenfold. DeepSeek trained a 671B-parameter V3, 37B active per token, on 2,048 H800 GPUs whose NVLink had been cut from 900 GB/s to 400 GB/s. This ISCA 2025 paper skips the architecture recap already in the technical report. It asks how hardware constraints should reshape the model, and what the next chip should add.
Each H800 node gets eight 400G InfiniBand NICs to offset the missing intra-node bandwidth. Almost every design choice in the paper tracks that 4:1 gap between NVLink and IB.
Three levers: store less, compute less, wait less on the network.
MLA compresses all attention-head KVs into one latent vector and caches only that vector. Per token this is 70 KB in BF16, against 328 KB for Qwen2.5-72B and 516 KB for LLaMA-3.1-405B, both GQA models.
DeepSeekMoE keeps training cheap. V3 costs about 250 GFLOPS per token; a 72B dense model costs 394, a 405B dense model 2,448. Expert-parallel all-to-all rides IB; intra-node forwarding rides NVLink. The gate is node-limited: 256 routed experts sit in 8 node groups, and each token may touch at most 4 nodes, so IB sends once and NVLink fans out.
Training uses fine-grained FP8, 1x128 tiles on activations and 128x128 blocks on weights. Ablations on 16B and 230B DeepSeek-V2 stayed within 0.25% of BF16. At decode, a single-layer MTP head drafts extra tokens; the second-token accept rate is 80-90%, and generation speed rises about 1.8x.
The cluster fabric is an eight-plane two-layer fat tree instead of a three-layer tree. In principle two layers reach about 16k GPUs. Export rules left them with a bit over two thousand.
The memory ledger is blunt: 70 KB/token versus 328 KB and 516 KB. The FLOP ledger matches: 250 GFLOPS/token for 671B sparse, under the 394 of a 72B dense model.
Decode is network-bound. With one expert per GPU, 32 tokens, and CX7 400 Gbps, two all-to-alls take 121 μs. Dual micro-batch overlap across 61 layers yields a 14.8 ms TPOT ceiling, about 67 tokens/s. On a GB200 NVL72 at 900 GB/s unidirectional, the same math drops to 6.7 μs and a 0.82 ms TPOT, about 1,200 tokens/s. The paper flags this as an ideal bound; small batches collapse GPU efficiency.
Multi-plane and multi-rail fat trees match on NCCL all-to-all. Training V3 on 2,048 GPUs, the two topologies sit inside measurement noise. CPU-side 64B latency: IB 2.8 μs same-leaf and 3.7 μs cross-leaf, against RoCE at 3.6 and 5.6 μs.
| Setup | Metric | Number |
| V3 MLA vs Qwen2.5-72B GQA | KV/token | 70 KB vs 328 KB |
| V3 vs LLaMA-3.1-405B | KV/token | 70 KB vs 516 KB |
| V3 vs 72B dense | Train FLOPs/token | 250 vs 394 GFLOPS |
| MTP | Decode speed | 1.8x |
| FP8 vs BF16 on V2 16B/230B | Relative loss | <0.25% |
This is a procurement memo as much as a model paper. MLA says long context is a KV problem, not a weight problem. Node-limited routing says expert count has to be designed against the intra/inter-node bandwidth ratio. FP8 trains, if you bring fine-grained scales and extra accumulation; Hopper Tensor Cores keep 13 mantissa bits and accumulate into an FP22 register.
For serving, MTP's 1.8x is cheaper than another NIC. For silicon, the ask is concrete: Tensor Cores that ingest group scales, a unified scale-up/scale-out programming model, and a communication coprocessor that takes forwarding, reduce, and casts off the SMs. Training already spends up to 20 of the H800's SMs on communication.
LogFMT maps activations into log space. At 8 bits it beats E4M3/E5M2; at 10 bits it matches BF16 combine. Fusing encode/decode with all-to-all costs 50-100% on Hopper, so they dropped it.
ConnectX-7 cannot do the ideal multi-plane NIC: cross-plane traffic still hairpins inside the node and adds inference latency. The 67 and 1,200 tokens/s figures assume compute fully hides communication; long contexts make MLA the critical path. FP8 ablations stop at 230B; there is no isolated 671B BF16 control. Node-limited routing ties the gate to the rack topology, so a new fabric means a new routing rule.