arXiv paper dissects 300 GitHub issues to find root causes of GPU kernel library failures in LLM systems

rohanpaul_ai · x · 2026-11-11

A new arXiv paper from Microsoft Research Asia and Tsinghua, "An Empirical Study of Issues in GPU Kernel Libraries for LLM Systems," presents the first comprehensive empirical study of failures in GPU kernel libraries underpinning LLM training and inference.

The authors manually analyzed 300 high-quality GitHub issues — with discussions and fixes — from seven representative libraries: FlashInfer, FlashAttention, Triton, CUTLASS, DeepGEMM, xFormers, and bitsandbytes, characterizing symptoms, root causes, fixes, and testing practices.

60.33% of sampled issues originate in the compiler/build system or runtime environment, so validation must cover the deployment and execution stack, not just kernel code.
Kernel specialization imposes implicit correctness constraints on tensor shapes, metadata layouts, memory aliasing, and hardware resources; violations require explicit contract validation and boundary-case testing.
Current testing and debugging practices are often insufficient for identifying low-precision numerical problems and hardware-specific execution failures.

The paper notes these failures can silently corrupt outputs, hang GPUs, block compilation, or degrade performance — wasting expensive GPU resources — and outlines research and tooling directions for future LLM GPU kernel development.

Original post →

arXiv paper dissects 300 GitHub issues to find root causes of GPU kernel library failures in LLM systems

More from Infra