Ex-evals developer: using training data samples as evals solved benchmaxxing but gave up too much
JoshPurtell · x · 2026-11-02
Josh Purtell reflects on a shift in eval design: there used to be excellent evals built purely for evaluation, which n-1 labs would inevitably train on and benchmaxx, drawing expected criticism. The industry "solved" this, he notes, largely by stopping purpose-built evals and instead using samples of training data as evaluations.
He argues that while training-data samples indeed can't be benchmaxxed, that misses the point: train/eval overlap shouldn't be a problem for the evaluation developer in the first place, and in solving it "we gave a lot up." An insider's critique of the current trend away from carefully designed benchmark sets.
More from Research
- SketchDirector: agentic layout reasoning generates complex scene sketches — GVCLab · 2026-11-02
- Thought experiment: would AI agents in a physics sim discover laws or reverse-engineer the engine? — zxpawanx · 2026-11-02
- HalachaBench sparks debate: is Talmudic Q&A testing knowledge or reasoning? — teortaxesTex · 2026-11-02
- Sebastian Raschka releases free 11-hour course on building reasoning models from scratch — AxSaucedo · 2026-11-02
- Microsoft and Berkeley open-source TokenEconomizer, cutting coding agent costs 18–31% — victor_explore · 2026-11-02
- TLA-Verus uses LLM agents to prove temporal properties of TLA+ specs — tianyin_xu · 2026-11-02