DeepSeek-V4-Pro training on Ascend 910C exposes long-tail kernel bottlenecks
teortaxesTex · x · 2026-07-23
The post points to a paper/table on DeepSeek-V4-Pro training on Huawei Ascend 910C and highlights one missing detail: it still says nothing about the 910C/950 question. The image shows a profiling table for training on Ascend, ranking key operators by task-duration share and utilization.
What the table says
- The top bottlenecks include SparseAttnSharedkvGrad, MatMulV3, and GroupedMatmul (MoE experts).
- The paper emphasizes a long tail of fragmented, memory-bound kernels.
- These small kernels are individually short, but together account for a substantial share of total compute time.
Why it matters
- This is a concrete look at where training efficiency is lost on Ascend NPU stacks.
- It suggests that not only the large matrix ops, but also many small memory-bound kernels, matter for overall utilization.
More from Infra
- Merge Gateway launches Fusion with $10 free credits and major-model support — shensi · 2026-07-23
- Google’s capex is on track to nearly triple in two years, sparking payback questions — SumitGup · 2026-07-23
- JADEPUFFER ransomware is targeting AI pipelines without using a zero-day — TechNadu · 2026-07-23
- Google TPU revenue may be around $1B to $1.5B, with outside customers to watch — BenBajarin · 2026-07-23
- Most of Google’s bought GPUs are still in warehouses, not datacenters — SumitGup · 2026-07-23
- Databricks and Microsoft extend their AI partnership through the 2030s — jefrankle · 2026-07-23