Anthropic's ex-Claude Code product lead: 10 great evals beat hundreds
aakashgupta · x · 2026-11-11
Aakash Gupta compiles first-hand eval advice from top model PMs: Cat Wu (ex-Claude Code product lead) says 10 great evals suffice to quantify goals and progress, alongside habits like asking the model to explain its own behavior and finding five trusted users. Dianne Penn starts from 30-40 failure examples; Daniel McKinnon targets evals the model passes 25-50% of the time. Gupta's own 48-question eval caught a version citing 38 nonexistent sources out of 47 — and flagged two grader errors. He found 76% of 46 open model PM roles require evals, and shares a four-step starter recipe.
- Small, hard, single-metric eval sets are the consensus
- Evals catch what vibe checks miss
More from coding & agent
- Comet Builds Docker-Based Incident Testbed to Evaluate Remediation Agents with Opik — craigsdennis · 2026-11-11
- Claude agent drives a real Leica microscope via MCP, autofocus and all — jwt0625 · 2026-11-11
- Yacine's agent talk: context moves outside the model, harnesses self-edit — yacinelearning · 2026-11-11
- UluP Nexus: a read-only MCP server for discovering indie software with verified submissions — Potential-Art7696 · 2026-11-11
- Cursor ships multiplayer in Projects: shared task board with agent-to-agent requests — mattyp · 2026-11-11
- Why agents will soon have only one tool: Code Mode cuts spend 15%, +3.5 Terminal-Bench points — dl_weekly · 2026-11-11