Josh Purtell flags high variance risk in eval sets with just ~10 tasks per domain
JoshPurtell · x · 2026-11-02
Josh Purtell argues that optimization pressure imposes demands on training data that evals simply don't face — but that doesn't mean eval work is easy; you have to work harder elsewhere.
Concretely, with only 10 tasks per domain/split, he worries that high-quality tasks at such small dataset sizes may have higher variance and lower bias than ideal, especially given the tasks are created by people whose day job is sourcing training data.
More from Research
- SketchDirector: agentic layout reasoning generates complex scene sketches — GVCLab · 2026-11-02
- Thought experiment: would AI agents in a physics sim discover laws or reverse-engineer the engine? — zxpawanx · 2026-11-02
- HalachaBench sparks debate: is Talmudic Q&A testing knowledge or reasoning? — teortaxesTex · 2026-11-02
- Sebastian Raschka releases free 11-hour course on building reasoning models from scratch — AxSaucedo · 2026-11-02
- Microsoft and Berkeley open-source TokenEconomizer, cutting coding agent costs 18–31% — victor_explore · 2026-11-02
- TLA-Verus uses LLM agents to prove temporal properties of TLA+ specs — tianyin_xu · 2026-11-02