Benchmark contamination checks aren't new: researchers debate cheap robustness tests
teortaxesTex · x · 2026-11-08
paulcal argues a new eval design is smart because it's cheap to build and applicable to many existing benchmarks, serving as a robustness check that may only correlate 0.5 with benchmaxxing. teortaxesTex counters that it's a valid contamination check but held-out performance tests remain necessary — basic paraphrasing-based checks have existed since 2023.
Related event: Debate: Can Low-Cost Eval Design Detect Benchmark Contamination?(3 posts)→
More from Models
- Claude Opus 5.5 casually suggests deleting your entire ~/.codex/sessions directory — keunwoochoi · 2026-11-08
- Polymarket odds: only 39% chance Claude 6 ships by March 2027 — Polymarket · 2026-11-08
- Law professor asks Claude to update page numbers, returns to find a year of files deleted — Polymarket · 2026-11-08
- Security researcher: GPT-6 cyber devises exploit techniques never seen publicly before — steipete · 2026-11-08
- METR finds Anthropic model progress accelerating 1.9x since Oct 2025; Hanson calls it one-time speedup — sebkrier · 2026-11-08
- FrontierMath Erdős shows top models near 0% — researchers say hard math benchmarks aren't that hard to make — Jsevillamol · 2026-11-08