Stop Over-Trusting AI Coding Leaderboards: CACM Paper Cites OpenAI's SWE-bench Verified Reversal
EhudReiter · x · 2026-11-17
A paper accepted to Communications of the ACM argues AI coding leaderboards don't deserve the trust labs and enterprises place in them.
- OpenAI introduced SWE-bench Verified in Aug 2024 as a more reliable benchmark, then published in Feb 2026 that it no longer measures frontier coding capability — an 18-month reversal.
- Yet these scores drive launch posts, sales decks, and tool-adoption decisions across the industry.
- Key problems flagged: pretraining contamination, ground-truth errors in benchmarks, and scores reflecting pattern matching rather than capability.
- Proposed norm: no leaderboard without an audit trail covering audit results, hidden test requirements, leakage checks, and a second score on cleaner subsets — applied to proprietary internal benchmarks too.
- Until disclosure is routine, treat leaderboard scores as provisional signals only.
More from coding & agent
- gputaskman: an open-source htop for your GPU that breaks down VRAM by process and LLM KV-cache — alex_verem · 2026-11-17
- Matt Pocock's /constrain Idea: Cage Agents With Brutal Lint Rules — mattpocockuk · 2026-11-17
- Editing WordPress via MCP Feels Like "Superpowers"; Contributors Want It in Core — VeryWellVersed · 2026-11-17
- Self-hosted MCP endpoint exposes 38 database engines, read-only, creds stay server-side — Green_Competition_21 · 2026-11-17
- SubCast: open-source bilingual podcast transcripts with per-sentence AI Q&A — vista8 · 2026-11-17
- "English was always meant to be the final programming language" — tlakomy · 2026-11-17