Benchmark 'Cheating' Debate: Commercial Endpoints Should Be Judged by What Works

RexDouglass · x · 2026-11-17

Pushing back on benchmark 'cheating' discourse, the author argues: if there is a specific private benchmark with intentionally generated, known out-of-sample text and vendors report on it, link it. If scoring is done on commercial endpoints, then it's whatever works — likely an ensemble of cheap and expensive signals covering different corpora and price/latency points, so he doesn't understand the fanboying either way.

Related event: Researchers Debate Whether Pangram's Benchmark Results Reflect Cheating(6 posts)→

Original post →

More from Models

Models channel →