Anthropic adds build-eval and hillclimb commands to automate eval design with Claude
RLanceMartin · x · 2026-09-29
Anthropic's Lance Martin published a guide on designing evals and hillclimbing without fooling yourself. The claude-api skill now offers /claude-api build-eval to create evals inside your codebase and /claude-api hillclimb to improve your app one change at a time, with held-out examples to catch overfitting. Key principles: tasks should mirror production; stronger models and more thinking should score higher (if not, tasks or graders are miscalibrated); and the frontier model at max effort should sit well below 100% so improvements remain measurable.
More from coding & agent
- Baseten's Guide: Evaluate Open Models for Coding Agents on Your Own PRs — baseten · 2026-11-16
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16