When tasks can't be sorted by difficulty, split your eval into multiple evals
JoshPurtell · x · 2026-11-02
Continuing the thread: when a domain measures many different tasks and tasks can't be sorted by difficulty, the basic fix is splitting into multiple evals. Where that's not possible or attractive, Josh Purtell still argues the goal should stand — knowing how good a model is from only a few tasks.
More from coding & agent
- OpenAI launches ChatGPT Sites: visitor login, built-in DB, storage and analytics — gabrielchua · 2026-11-02
- ChatGPT Sites packs login, built-in database, analytics and MCP hosting — gabrielchua · 2026-11-02
- Pain points of deploying AI agents: guardrails too tight slow automation, too loose invite risk — bibinprasannan · 2026-11-02
- Microsoft and Berkeley open-source TokenEconomizer, cutting coding agent costs 18–31% — victor_explore · 2026-11-02
- TLA-Verus uses LLM agents to prove temporal properties of TLA+ specs — tianyin_xu · 2026-11-02
- AI-generated App Store screenshots containing 'iPhone' trigger Guideline 5.2.5 rejection — tinyfool · 2026-11-02