Anthropic open-sources its eval design and hillclimbing playbook: 78.6%→90.5% accuracy at 1/5 cost
AGI Hunt · wechat · 2026-09-30
Anthropic's Lance Martin published a long-form post on claude.dev on designing evals and hillclimbing against them without fooling yourself. The method ships in the official claude-api skill via two new commands: /claude-api:build-eval and /claude-api:hillclimb.
- Four traits of a good eval: tasks sampled from production; stronger models/higher effort should score higher; the best model at max effort should still sit well below 100%; run-to-run variance must be small
- Adversarial sampling: don't pick cases just because today's model fails them — that measures the model's failure fingerprint, not real difficulty
- build-eval workflow: samples from production transcripts, bug reports, hand-written and synthetic cases; scorers prefer programmatic checks, falling back to LLM-as-judge (judge must not be the model under test)
- Overfitting guards: split eval into train/test; revert patches when train rises but test doesn't; never paste failures into prompts; structurally block reward hacking
- Result one: on a 44-ticket support benchmark, hillclimbing from Opus 4.8 high-effort down to Sonnet 5 low-effort with improved prompts lifted held-out accuracy from 78.6% to 90.5% at roughly one-fifth the cost
- Result two: the claude-api skill's own eval pass rate rose from 66% to 88%, uncovering flawed test items and a buggy scorer along the way
More from coding & agent
- Baseten's Guide: Evaluate Open Models for Coding Agents on Your Own PRs — baseten · 2026-11-16
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16