Anthropic launches plugin evals to check if Claude skills still work with new models
omarsar0 · x · 2026-09-12
Anthropic shipped plugin evals to address a common pain point: it's hard to know if your Claude skills still work after new model releases. Running claude plugin eval init inside a plugin folder sets up evaluation. Elon Omarsar welcomed the move and asked about open-sourcing it, arguing community input could make an evals plugin far more useful; he maintains his own meta harness but finds it hard to keep skills effective as models keep iterating.
Related event: Claude Code Launches plugin eval to Quantify Plugin Value(10 posts)→
More from coding & agent
- Microsoft Agensh board: a coding agent's "tests passed" is just a message — shoumikgoswami · 2026-10-28
- Ollama and FriendliAI to Host Seoul Meetup on Building Production Agents with Open Models — ollama · 2026-10-28
- theo: If Your Product Requires a Dashboard, Claude Will Just Rebuild It — sull · 2026-10-28
- Magic Patterns' first engineer: Cursor in Slack, a Patterson review agent, repo clones over worktrees — Y Combinator Blog · 2026-10-28
- Dev builds AI job-search app for his wife, she lands a job, and he open-sources it — CurieuxExplorer · 2026-10-28
- Supercharging LLM classification with logprobs: soft classifiers you can threshold-tune — JnBrymn · 2026-10-28