Semi-private evals with holdout sets are 'basically dead' as vendors hillclimb
xeophon · x · 2026-08-19
@xeophon argues that semi-private evals and evals with holdout sets are basically dead: data vendors will simply build synthetic environments to hillclimb the benchmarks. He sees only two viable paths going forward — fully open-source evals (e.g. MazeBench) or internal evals (e.g. Cursorbench). The take directly targets benchmark contamination through targeted optimization.
More from Models
- Opinion: Labs should restore Opus 3 and GPT-4 to boost enjoyment — natesiggard · 2026-08-19
- Qwen 72B model breaks the size-intelligence curve, resetting consumer hardware expectations — Hesamation · 2026-08-19
- Nori V1: 30M Param Open-Source Model Beats XGBoost Zero-Shot — rohanpaul_ai · 2026-08-19
- Open-Source Multi-Label Emotion Classifier Released — soumitrashukla9 · 2026-08-19
- Professor tests AIs on an open conjecture — all agree a general proof remains elusive — lemire · 2026-08-19
- Asking Claude to summarize a code lib requires understanding it first, defeating the point — cjimti · 2026-08-19