300 failure cases: 60% of GPU kernel bugs come from toolchain, not kernel code

青稞AI · wechat · 2026-11-20

Researchers from Microsoft Research Asia and Tsinghua systematically analyzed 300 high-quality GitHub failure cases across seven mainstream GPU kernel libraries: FlashInfer, FlashAttention, Triton, CUTLASS, DeepGEMM, xFormers, and bitsandbytes.

Failure distribution: deployment failures 27.33%, build failures 24.33%, runtime crashes 21%, functional errors 17.67%, performance regressions 7.67%.

Key finding: 60.33% of root causes lie in compilers/build systems and the runtime environment — not the kernel code itself. Notable pitfalls include tensor shape/memory layout mismatches producing wrong outputs, low-precision numerical drift, and special dimensions hitting shared-memory limits.

The team recommends compatibility/integration testing, differential testing, boundary testing, dynamic analysis, and performance regression testing. Reliability is becoming a core part of performance as LLM systems increasingly depend on low-level operator libraries.

Original post →

More from Infra

Infra channel →