Claude Code Ships 'claude plugin eval' as Hamel Husain Details Four Key Shortcomings
sh_reya · x · 2026-09-12
ClaudeDevs shipped a new Claude Code feature, claude plugin eval: create test cases, run your plugin or skill against them, score the runs, then re-run each case without the plugin to measure its actual value.
Hamel Husain tried it and outlined improvements:
- In-situ feedback: the workflow quizzes you up front about your recollection of plugin use; better to collect feedback while you're actually using it
- Error discovery first: it auto-builds datasets and judges from upfront answers and docs; it should start by annotating real traces/session history with you to find what's broken
- Clunky setup: configuring it for a skill that wasn't a plugin took 10 minutes of asking Claude
- Report readability: the generated HTML takes a long stare to decode
Related event: Claude Code Launches plugin eval to Quantify Plugin Value(10 posts)→
More from coding & agent
- Microsoft Agensh board: a coding agent's "tests passed" is just a message — shoumikgoswami · 2026-10-28
- Ollama and FriendliAI to Host Seoul Meetup on Building Production Agents with Open Models — ollama · 2026-10-28
- theo: If Your Product Requires a Dashboard, Claude Will Just Rebuild It — sull · 2026-10-28
- Magic Patterns' first engineer: Cursor in Slack, a Patterson review agent, repo clones over worktrees — Y Combinator Blog · 2026-10-28
- Dev builds AI job-search app for his wife, she lands a job, and he open-sources it — CurieuxExplorer · 2026-10-28
- Supercharging LLM classification with logprobs: soft classifiers you can threshold-tune — JnBrymn · 2026-10-28