Ex-evals developer lays out why training data and eval data demand different standards

On November 2, Josh Purtell posted a series of threads raising methodological questions about the design of a certain evaluation set. His core argument: data prepared for RL training and data prepared for model evaluation require two different standards of rigor, and the two should not be conflated.

Confirmed

• Purtell argues there is an inherent tension between benchmark design and RL training data design: in training settings, variance is a good thing and is expected, helping yield clean policy gradients; in evaluation settings, variance complicates analysis unless sample sizes are very large—and large samples bring problems of their own. He frames this tension as the classic bias-variance tradeoff.
• He illustrates an ideal evaluation shape: across 100 tasks, each task's pass rate should follow a sigmoid/step function with model capability—once capability reaches a certain level, the task is consistently passed, so pass@1 scores grow steadily with capability, making it easy to pinpoint capability levels.
• He questions the specific evaluation set under discussion: each domain/data split contains only about 10 tasks, and the task creators are themselves buyers of training data; even if high quality, these tasks on a small dataset may exhibit a "high-variance, low-bias" profile, making evaluation results hard to analyze.

Why it matters

• The discussion highlights a common but often overlooked problem in current AI evaluation: mixing the rigor standards of training data and evaluation data can blur benchmark conclusions and make them hard to interpret.
• Small-sample, high-variance evaluation sets may fail to reliably distinguish differences in model capability even when the tasks themselves are of high quality—directly affecting the credibility of evaluation results and how leaderboards should be read.

2026-11-02 ~ 2026-11-02 · 17 related posts

• [source] Benchmark rigor is getting muddled between training-data and eval-data design — JoshPurtell · 2026-11-02
• Eval tasks should be step functions: Josh Purtell on the tension between RL data design and benchmarks — JoshPurtell · 2026-11-02
• Evals want step-function tasks, training wants smooth pass@k curves — why the two goals conflict — JoshPurtell · 2026-11-02
• Eval task design is a bias vs variance tradeoff, argues eval engineer — JoshPurtell · 2026-11-02
• Evals With ~10 Tasks Per Domain May Trade Bias for Unhelpful Variance, Critic Warns — JoshPurtell · 2026-11-02
• Josh Purtell flags high variance risk in eval sets with just ~10 tasks per domain — JoshPurtell · 2026-11-02
• Reward Hacks Devastate RL Training but Barely Matter in Evals, Says Engineer — JoshPurtell · 2026-11-02
• CritPt v1 Had Largely Invalid Data Yet Worked Fine as an Eval Until Saturated — JoshPurtell · 2026-11-02
• Training data must be valid under all searched policies, eval data doesn't — JoshPurtell · 2026-11-02
• Why vocab tests work for human IQ but Goodhart in model training — JoshPurtell · 2026-11-02
• [source] Best evals must be Goodhart-able: they only sample a snapshot of capability — JoshPurtell · 2026-11-02
• The industry "solved" benchmark gaming by simply stopping making evals — JoshPurtell · 2026-11-02
• Ex-evals developer: using training data samples as evals solved benchmaxxing but gave up too much — JoshPurtell · 2026-11-02
• Evals: how few tasks do you need to judge a new model? — JoshPurtell · 2026-11-02
• When tasks can't be sorted by difficulty, split your eval into multiple evals — JoshPurtell · 2026-11-02
• Dev's AGI eval: a system that earns $1M/year profit from one prompt a month is AGI — JoshPurtell · 2026-11-02

1 near-duplicate retellings: JoshPurtell