Stanford's ParallelKernelBench lands NeurIPS 2026: LLMs nail only 32% of multi-GPU kernels

simonguozirui · x · 2026-11-04

Stanford and UCSD researchers announced ParallelKernelBench (PKB), a benchmark for LLM-generated multi-GPU kernels, accepted at NeurIPS 2026.

Multi-GPU kernel generation is uniquely hard: a combinatorially large design space across tensor/expert/pipeline/data/sequence parallelism, single-GPU roofline analysis misses communication bottlenecks, and communication paths (copy engine, TMA, SM instructions) carry distinct tradeoffs.
PKB spans 87 problems covering parallel AI workloads.
Results: zero-shot, the best model solves only 32% of problems, with just 25% faster than the unoverlapped PyTorch + NCCL baseline.
They propose a communication-aware roofline, finding 90%+ of baseline kernels reach under 50% of peak hardware utilization; some LLM-generated kernels are net-new and outperform existing ones.

More from Infra