Demystifying Agent Skills: Why They Work-Until They Don't
Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao, Rui Cao, Mengdi Wang, Shilong Liu, Yijiang Li
cs.AI
2026-08-14
Skills help mainly by anchoring procedure (65.7%), not injecting facts (4.5%); on identical trajectories they beat Workflow Memory by 6.06 points.
Agent skills are turning into standard infrastructure: Anthropic runs a public skill repository, and agent frameworks increasingly let models write and load reusable SKILL.md files. Existing evaluations answer one question only, whether adding a skill raises aggregate task success. Nobody could say which part of a skill does the work, which failures it repairs, or when it backfires, so skill authoring remains heuristic trial and error.
A team from Princeton, Stanford, USC, UC San Diego and Johns Hopkins ran controlled experiments plus human-validated trajectory coding to answer three questions: when skills help, why they work, and where they fail.
Three arms per task: Raw (no prior experience), Workflow Memory (cleaned procedural traces), and Skill (a standardized SKILL.md distilled from those same traces). The trajectory pool ranges from success-only to failure-only; both representations are built from identical trajectories and evaluated on the same tasks, so representation form is the only variable. Two agent pairings, Codex with GPT-5.3-Codex and Gemini CLI with Gemini-3.1-Pro-Preview, run across Terminal-Bench 2, Terminal-Bench Pro and SkillsBench.
For mechanism attribution, 8,135 trial records were normalized, 240 trajectories open-coded, and 238 valid labels consolidated into 12 modes under three categories. Human validation passed all 714 trajectory-label checks and agreed with LLM aggregation on 95.8% of labels (Cohen's kappa = 0.952). 528 paired triples compare all three arms on the same task. A separate retrieval study grows skill pools from 5 to 100 with random, similar, or dissimilar distractors, measured under embedding ranking, explicit agent selection, and full execution, with no outputs passed between the three.
| Mechanism | Share of skill-arm cases |
| Procedural anchoring | 65.7% |
| Knowledge injection | 4.5% |
Oracle-status success lands at 61.9% for Skill, 59.1% for Raw, and 55.9% for Workflow Memory. The most robust gap is Skill over Workflow Memory at +6.06 points (95% bootstrap CI +0.76 to +11.36); since both come from the same trajectories, the difference is attributable to representation alone.
The failure structure shifts concretely. Environment infrastructure failures drop from 5.3% (Raw) to 0.2% (Skill), output-format mismatches from 7.4% to 3.2%, background service failures from 2.7% to 0.8%. Algorithmic logic errors (7.4%) and static verification without runtime checks (11.7%) barely move: skills stabilize execution, they do not repair a wrong solution. A new failure surface appears too, with skills misapplied or ignored in 10.0% of skill-arm cases versus 0.8% for Raw, while Workflow Memory pays in timeouts (10.6% versus 4.4% for Skill) because raw traces carry detours and failed branches.
Retrieval tells a separate story. As pools grow from 5 to 100, precision of ground-truth skill use during execution falls from 29.6% to 3.3%, while downstream success edges up from 36.4% to 39.3%; recall at pool size 100 stays at 54.3-73.6%. Offline, embedding top-1 precision slides from 88.3% to 76.9%, with semantically similar distractors as the main stressor (70.5% down to 53.4%). Selecting the exact right skill is neither sufficient nor necessary. Withholding success/failure labels during skill creation also hurts once failed trajectories enter the pool: 0.7462 versus 0.4000 for Gemini at a 3-success/2-failure mix.
Three usable conclusions. Write skills as procedures, ordered steps, checks, and tool sequences; do not expect them to supply missing facts. The token budget works out: on a matched 83-task set, Skill beats Raw by 5.5 points while spending 34.2K fewer tokens, and beats Workflow Memory by 4.8 points for 95.3K more. And since retrieval precision collapses while success stays flat, both evaluation and system design should drop the assumption that the annotated ground-truth skill must be invoked.
The authors list three: coverage is limited to terminal and tool-use benchmarks; only a few agent-model configurations were tested; the taxonomy comes from roughly 3% of records, so rare modes may be underrepresented. Two more caveats from the reading: the RQ4 Codex side switched to GPT-5.4 after GPT-5.3-Codex became unavailable, so those numbers only support within-pairing comparisons; and the lightweight-baseline check (Skill 79.2% versus Workflow Memory 62.3%) ran on just 26 Terminal-Bench-2 tasks.