Pangram benchmark-cheating debate: search layer would only add cost, dev argues

jonaskreutz01 · x · 2026-11-17

Part of a debate over whether the AI-detection model Pangram games benchmarks via a retrieval gate. The author argues most real-world calls involve brand-new papers and filtering Twitter/Reddit feeds, where search adds nothing but cost — so a search layer tuned purely to pass old-text benchmarks has little motive. He adds that a "good model plus a benchmark-only gate" hypothesis is possible but unfalsifiable, and that the simpler explanation is the model just works.

Related event: Researchers Debate Whether Pangram's Benchmark Results Reflect Cheating(6 posts)→

Original post →