Mutation Analysis Shows GPU-Kernel Benchmarks Miss 16.9% of Faults
NationalUniversityofSingapore · hf · 2026-09-22
Benchmarks for LLM-generated GPU kernels decide correctness with a few random inputs and loose float tolerances, and their verdicts feed leaderboards and RL rewards. This work introduces mutation analysis as an adequacy metric for kernel-benchmark oracles: 10,303 compilable faults injected into verified CUDA implementations of 188 KernelBench problems (7,384 with independent kill witnesses), scoring any test protocol by detection rate.
Key findings:
- The official check deterministically misses 16.9% of witnessed faults; only 8.7% of arithmetic faults escape, but 78.6% of precision faults do
- Mechanisms are explainable: a tolerance blind band grows with reduction size; input aggressiveness has a ceiling set by legitimate float variance
- Audits KernelBench-Verified: its gain splits into +4.0 from hidden inputs and +4.5 from tighter tolerance; exposes a published fuzzing recipe that falsely rejects correct kernels 107 times
- Kill-matrix-optimized suites hit 98.0% detection with just two inputs per problem (94.8% held-out)
- Blindness grows with scale across 48 architectures; two problems prove unrefereeable — their official references violate the benchmark's own tolerance
Released as the KernelBench-M dataset.
More from Research
- Qonto open-sources QontoFAQ, a retrieval benchmark closer to real product Q&A — espadrine · 2026-09-22
- Podcast: Epoch AI researcher on RSI, robotics, China model gap, and open vs closed safety — natolambert · 2026-09-22
- Xiaomi's Luo Fuli Recaps MiMo-V2.6 RL Run: 25K Agent Trajectories Per Step — bookwormengr · 2026-09-22
- Winning robotics: deploy early and lean on collaborative perception instead of chasing reliability — broodsugar · 2026-09-22
- François Fleuret: math and CS modularity explains AI's IMO success — francoisfleuret · 2026-09-22
- Experiment suggests modern LLMs like Qwen hide tiny GPT2 self-models inside — paraschopra · 2026-09-22