KernelBench-Verified: no frontier model beats PyTorch when evals get strict, Meta/Stanford find
lmoroney · x · 2026-10-03
Meta and Stanford researchers released KernelBench-Verified, a much stricter evaluation of LLM-generated GPU kernels (open-sourced at facebookresearch/kernelbenchverified).
Stricter protocol:
- TF32-enabled PyTorch baseline matching what practitioners run on modern GPUs.
- A four-distribution hidden correctness suite to catch reward hacks, plus peak-memory tracking.
Results:
- Under this protocol, none of the seven frontier models tested beat PyTorch on average.
- The best, GPT-5.5, achieved 0.88x geometric-mean speedup, versus 1.43x under the looser original protocol — showing how much baseline and test design shape 'speedup' claims.
Takeaway: treat every speedup claim as a systems question — what baseline, what tests, does it still win in real practice?
More from Models
- SGLang ships Day-0 serving support for new model DaisyRose-27B — ying11231 · 2026-11-19
- Tibo's spicy take on why Astra 6.2 is getting such bad reviews — eigenrobot · 2026-11-19
- User builds interactive 3D molecular viewer macOS app with GPT-6.2 Astra — DeryaTR_ · 2026-11-19
- Caliper Harness: open-source coding harness disciplines small local models with plans, checks, reviews — Saraozte01 · 2026-11-19
- DaisyRose-27B's in-house sandbox: ~80% cheaper than cloud providers, 99.95% success rate — BanghuaZ · 2026-11-19
- Tencent's 220B Hy4-flash quantized to 49 GB weights, runnable on 64 GB VRAM — gladkos · 2026-11-19