Evals: how few tasks do you need to judge a new model?
JoshPurtell · x · 2026-11-02
Josh Purtell frames a core eval-design question: how many tasks must you run to judge a new model? In domains with few atomic skills, you can sort tasks by the threshold they're sensitive to and binary-search quickly. When a domain spans many skills, split it into multiple evals instead. He also criticizes using training-data samples as evals — it prevents benchmaxxing but shifts a solvable problem back onto eval developers at great cost.
More from coding & agent
- OpenAI launches ChatGPT Sites: visitor login, built-in DB, storage and analytics — gabrielchua · 2026-11-02
- ChatGPT Sites packs login, built-in database, analytics and MCP hosting — gabrielchua · 2026-11-02
- Pain points of deploying AI agents: guardrails too tight slow automation, too loose invite risk — bibinprasannan · 2026-11-02
- Microsoft and Berkeley open-source TokenEconomizer, cutting coding agent costs 18–31% — victor_explore · 2026-11-02
- TLA-Verus uses LLM agents to prove temporal properties of TLA+ specs — tianyin_xu · 2026-11-02
- AI-generated App Store screenshots containing 'iPhone' trigger Guideline 5.2.5 rejection — tinyfool · 2026-11-02