Semi-private evals with holdout sets are 'basically dead' as vendors hillclimb
xeophon · x · 2026-08-19
@xeophon argues that semi-private evals and evals with holdout sets are basically dead: data vendors will simply build synthetic environments to hillclimb the benchmarks. He sees only two viable paths going forward — fully open-source evals (e.g. MazeBench) or internal evals (e.g. Cursorbench). The take directly targets benchmark contamination through targeted optimization.
More from Models
- Meta open-sources Muse Spark Open agent model with day-0 TPU support in SGLang — BanghuaZ · 2026-10-16
- Saying "load-bearing" in your prompt makes Opus 5.5 stop holding back — zacharynado · 2026-10-16
- Americans get their Claude accounts banned while on business trips in China — dotey · 2026-10-16
- I priced every Grok Bot call: $20 to $200 plan in 5 days, long jobs eat 52% — jonathanmalkin · 2026-10-16
- NVIDIA releases Nemotron-3.5-Embed-1B, a multimodal embedding model covering 34 languages — jacek2023 · 2026-10-16
- EpochAI pegs GPT-6 training at up to 1.8e27 FLOP; scaling01 calls param estimates way too high — scaling01 · 2026-10-16