DeepSeek-V4-Pro training on Ascend 910C exposes long-tail kernel bottlenecks
teortaxesTex · x · 2026-07-23
The post points to a paper/table on DeepSeek-V4-Pro training on Huawei Ascend 910C and highlights one missing detail: it still says nothing about the 910C/950 question. The image shows a profiling table for training on Ascend, ranking key operators by task-duration share and utilization.
What the table says
- The top bottlenecks include SparseAttnSharedkvGrad, MatMulV3, and GroupedMatmul (MoE experts).
- The paper emphasizes a long tail of fragmented, memory-bound kernels.
- These small kernels are individually short, but together account for a substantial share of total compute time.
Why it matters
- This is a concrete look at where training efficiency is lost on Ascend NPU stacks.
- It suggests that not only the large matrix ops, but also many small memory-bound kernels, matter for overall utilization.
More from Infra
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
- Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11