Microsoft & Tsinghua: 60% of GPU kernel library failures in LLM systems come from build and environment, not kernel code

rohanpaul_ai · x · 2026-11-11

A Microsoft Research Asia and Tsinghua paper presents the first comprehensive empirical study of GPU kernel library issues in LLM systems, manually analyzing 300 closed GitHub issues across seven libraries: FlashInfer, FlashAttention, Triton, CUTLASS, DeepGEMM, xFormers, and bitsandbytes.

Key findings:

60.33% of issues originate in the compiler/build system or runtime environment, with only 39.67% in the kernel library itself — validation must cover the deployment stack, not just kernel code.
The single largest cause is software incompatibility among drivers, CUDA, frameworks, and packages: 76 issues (25.33%).
Kernel specialization imposes implicit correctness constraints on tensor shapes, metadata layouts, memory aliasing, and hardware resources; violations can silently corrupt outputs even for supported operations.
Current testing and debugging practices are often insufficient for low-precision numerical issues and hardware-specific failures.

The authors argue these failures can silently corrupt tokens, hang GPUs, and waste expensive compute, and outline research directions including explicit contract validation and boundary-case testing.

Original post →

Microsoft & Tsinghua: 60% of GPU kernel library failures in LLM systems come from build and environment, not kernel code

More from Infra