Eval tasks should be step functions: Josh Purtell on the tension between RL data design and benchmarks

JoshPurtell · x · 2026-11-02

Josh Purtell argues there's a real tension between benchmark design and RL training data design. Ideally, each of 100 eval tasks should behave like a sigmoid/step function, so pass@1 scores rise consistently with model capability. He calls this a synecdoche for how two different standards of rigor — data design for clean policy gradients vs. data design for interpretable evaluation — have gotten muddled since they apply to roughly the same environments.

Original post →

Eval tasks should be step functions: Josh Purtell on the tension between RL data design and benchmarks

More from coding & agent