Claude Code plugin evals: real model calls, trust warnings, and CI gating explained
ClaudeDevs · x · 2026-09-12
Anthropic's docs detail how claude plugin eval works: every run and judge grader is a real model call counted against your plan or API bill, so pilot with --runs 1 first; plugins' hooks and MCP servers run as you, so only evaluate plugins you trust. Each case combines a realistic prompt with graders (regex, tool-call checks, or model-judged rubrics). Use evals to measure how reliably your plugin steers Claude, catch regressions, compare against a no-plugin baseline, and gate CI on scores. Try it with claude update.
Related event: Claude Code Launches plugin eval to Quantify Plugin Value(10 posts)→
More from coding & agent
- The legendary fake Go generics: Unicode Canadian Syllabics as angle brackets — rickasaurus · 2026-10-28
- Microsoft Agensh board: a coding agent's "tests passed" is just a message — shoumikgoswami · 2026-10-28
- Ollama and FriendliAI to Host Seoul Meetup on Building Production Agents with Open Models — ollama · 2026-10-28
- theo: If Your Product Requires a Dashboard, Claude Will Just Rebuild It — sull · 2026-10-28
- Magic Patterns' first engineer: Cursor in Slack, a Patterson review agent, repo clones over worktrees — Y Combinator Blog · 2026-10-28
- Dev builds AI job-search app for his wife, she lands a job, and he open-sources it — CurieuxExplorer · 2026-10-28