Anthropic's New Guide: Claude Automates Eval Design and Hillclimbing, Cost Cut ~5x
新智元 · wechat · 2026-10-09
Anthropic released an official guide for automating the eval-and-tweak loop when building apps on Claude, so developers stop hand-testing every prompt change.
- /claude-apibuild-eval: turns vague requirements into a runnable test suite; cover common cases, not just failures; graders must be validated against human judgment.
- /claude-apihillclimb: Claude proposes one change per round, reruns evals, keeps improvements and reverts regressions; held-out "acceptance" questions guard against overfitting.
- Results: on an internal support-ticket eval, decision accuracy rose from 78.6% to 90.5% on 14 held-out tickets while model-call cost dropped to roughly 1/5 (small-sample, specific config).
- Both commands require Claude Code v2.1.259+; the tuning itself costs tokens, so set an experiment budget.
The guide closes by pointing developers back to insight-gathering, grader quality, and quality/cost tradeoffs — the parts AI can't automate.
More from coding & agent
- Claude Code Pushes Urgent Security Update Across All Installations — umeshonai · 2026-10-23
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23