Stanford's ParallelKernelBench: best LLM solves only 32% of multi-GPU kernel tasks

burny_tech · x · 2026-11-05

Stanford and UCSD researchers released ParallelKernelBench (PKB) on alphaXiv, the first benchmark for evaluating LLMs at writing fast multi-GPU kernels, accepted as a NeurIPS 2026 poster.

Key points:

Multi-GPU kernel generation was unexplored even though communication is now a dominant bottleneck in large-scale training/inference; challenges include a combinatorially large parallelization design space (tensor/expert/pipeline/data/sequence), single-GPU roofline analysis missing communication costs, and tradeoffs across communication paths (copy engine, TMA, SM instructions).
The benchmark covers 87 problems spanning a taxonomy of parallel AI workloads.
Results: zero-shot, the best model gets only 32% correct, with just 25% faster than the unoverlapped PyTorch + NCCL baseline.
A communication-aware roofline analysis shows over 90% of baseline kernels achieve less than 50% of peak hardware utilization.
Several optimized LLM-generated kernels are net new useful kernels that outperform existing ones.

Original post →

Stanford's ParallelKernelBench: best LLM solves only 32% of multi-GPU kernel tasks

More from Infra