skillEd open-sources an MCP server for validating agent-skill evals and diffing regression reports
MiserableFlatworm337 · reddit · 2026-10-23
The author released skillEd, an open-source evaluation toolkit with a local stdio MCP server for validating agent-skill evals and comparing policy-regression reports — e.g., a skill that applies refund eligibility or rounding rules, where changes should be judged by explicit cases rather than vibes.
- Start with skilled mcp --workspace <path>; validation, dry runs, report comparison, and workflow guidance ship by default; live evals require --allow-execution, with host approval controls still applying.
- Recommended loop: validate skill and cases, inspect the planned run, run permitted evals, revise, then diff before/after reports — comparing saved reports costs no model calls. Keep independent holdout cases beyond the examples a revision was written to fix.
- It provides evaluation evidence, not runtime enforcement: permission checks stay with the host, and it targets trusted local workspaces only.
Code and a worked refund example: github.com/DeadHead04/skillEd.
More from coding & agent
- NVIDIA unveils OpenShell Reference Architecture to set boundaries for enterprise AI agents — NicoVerderosa · 2026-10-23
- Most 'unauthorized' AI agents in US bank accounts were let in by customers themselves, per Axios — nptacek · 2026-10-23
- Personal AI agents become a capital-intensive race as cloud execution environments reshape cost structures — sujingshen · 2026-10-23
- LLMs make code so cheap that recursive rewrites could make verified software tractable — jd_pressman · 2026-10-23
- Hugging Face hands OpenEnv RL environments project to Linux Foundation for neutral governance — lmoroney · 2026-10-23
- Theo writes the definitive explainer on what an AI coding "harness" actually is — pvncher · 2026-10-23