Claude Code Launches plugin eval to Quantify Plugin Value

Anthropic ships claude plugin eval in Claude Code to quantify the real value of plugins and skills, and to address the pain point of skills silently breaking after new model releases. The feature centers on test-case scoring with a with/without-plugin baseline comparison, with results shown in the terminal and exported as an HTML report; official docs cover billing, security, and CI gating.

Confirmed

The core workflow creates test cases, runs them with the plugin, scores the outputs, then re-runs the same cases with the plugin disabled to compare differences (m1, m7, m8, m10)
Running claude plugin eval init in the plugin directory bootstraps the suite: the developer describes good/bad outputs and provides a few real prompts, Claude drafts test cases and checks, then runs a pilot (m2, m5, m9)
After execution, the terminal shows per-case scores with and without the plugin, and an HTML report with full details is generated; on supported accounts the report is also published as a private artifact (m3)
Docs note that evals call the real model each run, consuming plan quota or incurring API charges—try --runs 1 first—and flag security considerations around plugin hooks and MCP servers (m4)
Docs also describe wiring evals into CI as a gating step (m4)

Why it matters

Judging whether a plugin or skill works—and still works after a model update—previously relied on manual spot checks; plugin eval provides a quantifiable baseline-comparison method (m8, m9)
As a built-in official feature, it embeds evaluation directly into the Claude Code workflow, enabling automated regression checks on model or plugin updates via CI

2026-09-12 ~ 2026-09-12 · 10 related posts

Primary sources

Claude Code ships plugin evals to score your plugin against a no-plugin baseline — ClaudeDevs · 2026-09-12 03:56 (UTC+08:00)
Claude Code plugin evals: real model calls, trust warnings, and CI gating explained — ClaudeDevs · 2026-09-12 03:56 (UTC+08:00)
Claude Code ships plugin eval: test if your skills still work with new models — trq212 · 2026-09-12 05:58 (UTC+08:00)

[source] Claude Code ships plugin evals to score your plugin against a no-plugin baseline — ClaudeDevs · 2026-09-12
Claude Code's eval init drafts test cases and estimates full-run cost for you — ClaudeDevs · 2026-09-12
Claude plugin eval outputs per-case scores and an HTML report with baseline diff — ClaudeDevs · 2026-09-12
[source] Claude Code plugin evals: real model calls, trust warnings, and CI gating explained — ClaudeDevs · 2026-09-12
[source] Claude Code ships plugin eval: test if your skills still work with new models — trq212 · 2026-09-12
Claude Code Ships 'claude plugin eval' as Hamel Husain Details Four Key Shortcomings — sh_reya · 2026-09-12
Anthropic launches plugin evals to check if Claude skills still work with new models — omarsar0 · 2026-09-12

3 near-duplicate retellings: daniel_mac8 · devdigest · sidjustice_