Researcher Questions Pangram Benchmark: Commercial Endpoint Could Use Every Cost Trick
RexDouglass · x · 2026-11-17
RexDouglass argued in an X discussion that Pangram's benchmark challenge appears to run against its commercial endpoint, which "would absolutely use every trick it could to reduce cost and increase performance," leaving room for gaming. He asked for links to a narrower test of the specific model.
hexane360 countered that fuzzy search isn't helpful here unless the sole goal is benchmark cheating, and that ensembling cheap and expensive signals shouldn't cause memorization unless the cheap signals are dumb. RexDouglass added that fuzzy search over large corpora is a solved, cheap problem, while conceding Pangram's training budget was likely insufficient to memorize the entire corpus.
Related event: Researchers Debate Whether Pangram's Benchmark Results Reflect Cheating(6 posts)→
More from Models
- Quantized Opus 5.5 says 'open' instead of 'run' — quantization quirks again — ryunuck · 2026-11-17
- OpenAI urged to release rumored model allegedly cracking Navier–Stokes millennium problem — haider1 · 2026-11-17
- Verus Proof Index: new formal-verification benchmark evaluates 25 models' agentic proofs — fhuszar · 2026-11-17
- Asking ChatGPT and Claude to play NeurIPS reviewers — they reject the papers — burny_tech · 2026-11-17
- Benchmark cheating debate: scaling a 3B open model to ~60B explains the scores — RexDouglass · 2026-11-17
- Physician slams OpenEvidence's "lowering all-cause mortality" claim; Eric Topol amplifies — EricTopol · 2026-11-17