CritPt v1 Had Largely Invalid Data Yet Worked Fine as an Eval Until Saturated
JoshPurtell · x · 2026-11-02
Josh Purtell extends his argument that eval data and training data face conflicting demands: a huge fraction of CritPt v1's data was straight-up invalid, yet before the valid subset was saturated it remained a pretty good eval. Training RL on it, however, would have thrown a wrench in the works.
Together with his earlier point — that minor reward hacks are devastating for training but largely harmless in evaluations at small doses — he argues that optimization pressure imposes demands on training data that evals simply don't have.
More from Research
- SketchDirector: agentic layout reasoning generates complex scene sketches — GVCLab · 2026-11-02
- Thought experiment: would AI agents in a physics sim discover laws or reverse-engineer the engine? — zxpawanx · 2026-11-02
- HalachaBench sparks debate: is Talmudic Q&A testing knowledge or reasoning? — teortaxesTex · 2026-11-02
- Sebastian Raschka releases free 11-hour course on building reasoning models from scratch — AxSaucedo · 2026-11-02
- Microsoft and Berkeley open-source TokenEconomizer, cutting coding agent costs 18–31% — victor_explore · 2026-11-02
- TLA-Verus uses LLM agents to prove temporal properties of TLA+ specs — tianyin_xu · 2026-11-02