MetaKernelBench measures cross-DSL GPU kernel skill transfer, with lifts from -19% to +29%

encodelab · hf · 2026-10-12

Researchers from Westlake University, SJTU and Zhejiang University introduce MetaKernelBench, the first benchmark measuring the reuse value of experience retained by GPU kernel optimization agents.

Motivation: kernel agents increasingly store learnings in knowledge bases or distilled skills, but existing benchmarks only score each attempt's correctness and speed, not whether retained experience transfers.
Setup: 74 fused-subgraph problems across six families, each posed as a pair of CuTe DSL and TIRx variants differing only in DSL. The agent attempts one variant solo, distills its learnings into a natural-language skill, and that skill is the only extra input for a retry by the same model in the other DSL, compared under matched per-attempt budgets.
Results: across six models and both directions, paired lift over solo attempts ranges from -19% to +29%. Four models gain in both directions, yet regressions hit 16%–45% of problems in every model/direction. Outcomes track the source attempt's result relative to the target's solo attempt: +71% improvement when the source stands above, 54% regression when below.

The benchmark complements implementation-quality metrics by measuring same-problem cross-DSL knowledge transfer, showing skill reuse is not always beneficial.

Original post →

MetaKernelBench measures cross-DSL GPU kernel skill transfer, with lifts from -19% to +29%

More from coding & agent