Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
Christopher Kevin, Narendran Raghavan, Jean-Francois Puget, Roshni Malani, Meghana Puvvadi, Moshe Abramovitch, Mohit Gupta, Rama Akkiraju, Subodh Prabhu, Yogesh Dangi, Wei Luo, Seong Hee Lee
KDD 2026 Workshop on Enterp
cs.AI
2026-08-21
ACES pairs live agent runs with and without a skill. Mean composite Skill Lift is 0.2134 across 947 cases (72.8% positive); scan scores barely track runtime value.
Agent skills are now the usual extension layer for Claude Code, Codex, Cursor, and similar harnesses: a SKILL.md file, optional scripts, and references the agent loads on demand. Public catalogs already hold thousands of these packages. The deployment question for an enterprise team is narrower than whether the file looks well written. Does this package help a live agent finish the task under the same model, sandbox, and grading policy?
Today's gates scan the document. Structural checks, LLM-as-Judge rubrics, script linters, and security scanners all read SKILL.md and related files. None of them runs the skill. NVIDIA compares that to compiling with -Wall -Werror: a clean scan does not mean the program does the right thing. A skill can pass every check and still fail at runtime because the agent never discovers it, calls the wrong script, misreports a correct tool output, collides with a sibling skill, or silently regresses after a model update. Those failures are invisible from the file alone.
ACES (Agentic Continuous Evaluation of Skills) treats a skill as an executable agent artifact. The core protocol is a paired trial: same prompt, same harness, same model, same supporting skills, same scorer; only the target skill's presence changes. The difference is Skill Lift, the target's marginal contribution under that fixed setup.
Evaluation assets live with the skill, usually as evals.json: a user question, an expected skill, a reference answer, and free-form expectedbehavior strings such as "read SKILL.md before executing." Authors can add EVAL.md so their intent outranks any LLM-seeded cases, or attach BYOT/BYOG when the product already owns tasks and graders. ACES then only stages the paired protocol and aggregates.
Trajectories are normalized into ATIF (Agent Trajectory Interchange Format) so one grading layer covers multiple harnesses. The default suite has six metrics:
Composite Skill Lift averages the six metrics with equal 1/6 weight. The outcome-only view averages accuracy and goal accuracy. The baseline workspace is not empty: configured decoys or prerequisites sit in both arms, so "any skill exists" is not credited to the target. Isolation mode stages only the target skill (content contribution). Group mode adds decoys; the gap is the routing premium. NVIDIA SkillEvaluator is the open-source implementation.
On 145 real skills from internal enterprise repos and public catalogs, structural scores average 79.2 (median 79.8, σ=4.9). 94.5% pass the default 70-point gate; only 48.9% reach 80. The LLM-judge rubric passes 86.2% at the same threshold. The two scan scores barely agree: Spearman ρ=0.14, Pearson r=0.08. Frontmatter is mostly aspirational: 99.3% omit tools, 97.9% omit Limitations, 97.2% omit author. Scans catch authoring issues. They do not converge, and they never watch the agent.
The live layer is the actual contribution. The production analysis keeps 64 skills and four primary harnesses (OpenCode, Claude Code, Codex, Terminus-2). 58 of those skills yield 947 scored paired cases.
| metric | with skill | baseline | Skill Lift | share of cases with +lift |
| composite | 0.7460 | 0.5326 | 0.2134 | 72.8% |
| accuracy | 0.7760 | 0.6329 | 0.1431 | 37% |
| goalaccuracy | 0.6887 | 0.4720 | 0.217 | 57% |
| skillexecution | - | - | 0.3263 | 64% |
| behaviorcheck | - | - | 0.2983 | 56% |
| skillefficiency | - | - | 0.2758 | 41.7% |
| security | - | - | 0.020 | 6% |
Of 947 cases, 689 are positive, 171 zero, 87 negative. Process metrics move more than final-answer accuracy: discovery, routing, workflow following, and tool use, none of which a document scan can see. Skill efficiency has the third-largest mean lift but a positive lift in only 41.7% of cases; some runs trade efficiency for correctness or workflow completion.
Harness-level mean lifts: OpenCode 0.3611, Claude Code 0.2904, Codex 0.1264, Terminus-2 0.0896. These are matched deltas against each harness's own baseline, not a model ranking. Coverage is uneven (Claude Code 56 cells / 251 cases; OpenCode 34 / 211). Holding Codex fixed and swapping backends, GPT-5.5 raises the no-skill baseline enough that Skill Lift shrinks from about +0.09 on GPT-5.2 to about +0.05, while absolute scores stay high. Stronger models need the skill less.
Scan scores have no useful monotonic relationship with live lift: structural Spearman ρ=-0.0181, LLM-judge ρ=-0.0266.
A 25-variant routing stress test varies visible skills from 1 to 50. Mean overall lift stays 0.133-0.149 from 1 to 20 visible skills, while wall time goes from 258s to 451s. At 50, the with-skill pass rate falls to 0.55 and wall time hits 1,290s. Crowded workspaces belong in a release stress test, not in every edit.
For teams shipping skills as product, a tidy SKILL.md is not runtime evidence. ACES treats evals/ as the skill's tests/ directory. Structural scans run on every change; live paired trials are for release candidates, high-risk skills, or reviewer-requested diffs. Negative Skill Lift splits into two classes in the traces: the with-skill arm failed to score, or the skill actually steered the agent the wrong way. The second class is the one that should change the description and trigger vocabulary.
This is not another agent leaderboard. Terminal-Bench and SWE-bench score agents on fixed tasks. ACES scores an arbitrary skill on the author's own tasks. SkillsBench showed that skills help in aggregate; it does not grade a single skill artifact or plug into CI. If you already pile skills onto Claude Code or Codex, the useful artifact here is a review protocol.
It is a methodology paper. The 0.21 Skill Lift is not a new-algorithm gain. It is the average with-versus-without gap on this enterprise corpus. The claim that survives is narrower and more useful: scan gates are not a proxy for runtime value.
The corpus skews toward System Access, Deployment, Platform, and Data Infra. Troubleshooting and creative skills are almost absent, so the lift distribution should not be generalized there. Harness coverage is uneven. The headline 95% CI is a normal interval over paired-case deltas, not 947 independent skills; a skill-level bootstrap is [0.1898, 0.2350] and still positive. Of 201 production cells, 88 have two trial files; the median within-cell lift SD is 0.0319.
The paired design holds task, harness, model, scorer, and supporting skills fixed, which is stronger than a single live score. It still does not identify an environment-independent skill contribution. Adding a skill changes routing pressure and context allocation. Skill Lift is the marginal contribution under the declared workspace and baseline policy.
Live inter-judge agreement, human calibration, and judge-uncertainty are unmeasured. On the document rubric, three judges spread by about 1.5 points on a 0-10 scale; the live layer has no matching number. Several enterprise skills call VPN-only endpoints the container cannot reach, so absolute goalaccuracy on those skills is a lower bound. The static security scanner is an external integration, not a contribution of this paper. Timeouts are mixed with other no-score causes.
Tasks are author-owned. That keeps evaluation close to intent, and it also means a skill is graded on questions its authors wrote. Decoys, negative controls, and isolation versus group mode reduce that risk. The paper does not quantify author-question bias.