Evals With ~10 Tasks Per Domain May Trade Bias for Unhelpful Variance, Critic Warns
JoshPurtell · x · 2026-11-02
JoshPurtell critiques an eval design where each domain/split gets only 10 tasks created by training-data procurers: in evaluation, high variance complicates analysis unless sample sizes are huge, and large samples bring issues of their own. He argues you'd rather have biased low-variance tasks for evals than high-variance ones.
More from Research
- SketchDirector: agentic layout reasoning generates complex scene sketches — GVCLab · 2026-11-02
- Thought experiment: would AI agents in a physics sim discover laws or reverse-engineer the engine? — zxpawanx · 2026-11-02
- HalachaBench sparks debate: is Talmudic Q&A testing knowledge or reasoning? — teortaxesTex · 2026-11-02
- Sebastian Raschka releases free 11-hour course on building reasoning models from scratch — AxSaucedo · 2026-11-02
- Microsoft and Berkeley open-source TokenEconomizer, cutting coding agent costs 18–31% — victor_explore · 2026-11-02
- TLA-Verus uses LLM agents to prove temporal properties of TLA+ specs — tianyin_xu · 2026-11-02