DeepSeek-V4-Pro training on Ascend 910C exposes long-tail kernel bottlenecks
teortaxesTex · x · 2026-07-23
The post points to a paper/table on DeepSeek-V4-Pro training on Huawei Ascend 910C and highlights one missing detail: it still says nothing about the 910C/950 question. The image shows a profiling table for training on Ascend, ranking key operators by task-duration share and utilization.
What the table says
- The top bottlenecks include SparseAttnSharedkvGrad, MatMulV3, and GroupedMatmul (MoE experts).
- The paper emphasizes a long tail of fragmented, memory-bound kernels.
- These small kernels are individually short, but together account for a substantial share of total compute time.
Why it matters
- This is a concrete look at where training efficiency is lost on Ascend NPU stacks.
- It suggests that not only the large matrix ops, but also many small memory-bound kernels, matter for overall utilization.
More from Infra
- Michael Dell: Musk is "the most demanding customer there is" — Colossus servers built in ~6 weeks — elonmusk · 2026-10-27
- AI chip startup Tenstorrent raises $1.5B at $6.2B valuation, more than doubling — Beth_Kindig · 2026-10-27
- Anthropic discloses up to $120.5B SpaceX compute deal, ~$3.1B monthly at full capacity — XFreeze · 2026-10-27
- Satire: swapping Raft for an LLM-style decision model tanks write throughput 99.98% — irombie · 2026-10-27
- Cadence joins the domain-specific AI model camp for chip design — karlfreund · 2026-10-27
- Qdrant builds Universal I/O to run vector search mmap code on object storage — qdrant_engine · 2026-10-27