Five principles for curating LLM agent evals: diverse, realistic, hard, discriminative, high-quality
cwolferesearch · x · 2026-10-28
Cameron Wolfe breaks down five key factors for selecting evaluation tasks for LLM/agent benchmarks:
- Diverse: cover the full range of capabilities; structured taxonomies like GDPval (by sector, occupation, work activity) help measure coverage and find gaps.
- Realistic: source tasks from real usage — CursorBench uses real Cursor sessions, SWE-Lancer uses actual Upwork freelance engineering tasks.
- Difficult: avoid saturation; estimate difficulty via human experts, task properties, or model-based filtering by running agents and measuring success rates.
- Discriminative: hard tasks that everyone fails carry little signal; Item Response Theory (IRT) can select smaller, more informative eval sets based on difficulty and discriminative power.
- High-quality: reference solutions, instructions, and graders must be accurate; Terminal-Bench uses failure analysis plus human/LLM auditing of trajectories and grader outputs to catch grading errors.
Key takeaway: a bigger or harder benchmark is not necessarily better — tasks must be diverse, realistic, challenging, discriminative, and accurately graded.
Related event: Cameron Wolfe Outlines Principles for Reliable LLM and Agent Evaluation(2 posts)→
More from coding & agent
- Multimodal GraphRAG models missing and contradictory evidence for storm damage assessment — JeremyCMorgan · 2026-10-28
- Vibe coding in real life: 20 hours of personal-project work done in 2 — gleech · 2026-10-28
- LangChain open-sources Content Studio, a Slack design agent, and shares its harness design — BraceSproul · 2026-10-28
- "If AI can do it, humans shouldn't": team lets AI agent review PRs — Yuchenj_UW · 2026-10-28
- Sequoia, Citi and OpenAI launch No Limit: build a poker bot in 60 minutes, $100K grand prize — polynoamial · 2026-10-28
- Nyxis: a Lean framework with formally proven authorization safety for AI-written backends — udmrzn · 2026-10-28