GPU QR competition audit: winning kernel ran 107x faster yet failed on new inputs
kfountou · x · 2026-11-18
A new paper argues that as AI coding agents produce numerical software faster than experts can review it, additional evidence of correctness is needed. The authors audited a public GPU QR competition: the winning entry ran 107x faster than the PyTorch/NVIDIA QR baseline, yet all five leading entries — despite passing every accuracy test the competition ran — failed on new inputs from the competition's own generator under the same checker, while the baseline passed all. Heatmaps show 30–66% error rates on rank-deficient, banded, and mixed inputs. The paper proposes VERNA, an audit pipeline (recognize → test against error bounds → certify into a certificate of trust) paired with VSCL machine-checked error bounds, as a framework for trustworthy AI-written numerical kernels.
More from Research
- Liquid AI's Maxime Labonne on post-training: SFT, RL, MOPD and regression — ziv_ravid · 2026-11-18
- Self-inoculation cuts RL reward hacking from 82% to 20% across 11 environments — repligate · 2026-11-18
- Embodied First Exam: 1,090 embodied tasks show agents crumble outside privileged settings — ManlingLi_ · 2026-11-18
- GPT-6.2 Astra tops Mercor's OpenWorldEngineer benchmark at 67.2% — morqon · 2026-11-18
- Google details EmbeddingGemma 2: unified multimodal embeddings across text, image, audio, video — GlennCameronjr · 2026-11-18
- LatchBio launches NeuroscienceBench and a 13-benchmark leaderboard for biology AI agents — kenbwork · 2026-11-18