SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, Chujun Tao, Binxu Li, Xuandong Zhao, Hejia Geng, Xiaojun Wu, Junwei Zhou, Xiaokun Chen, Hanwen Xing, Yubo Li, Qunhong Zeng, Di Wang, Yuanli Wang, Roey Ben Chaim, Penghao Jiang, Haotian Shen, Luyang Kong, Xinyi Liu, Runhui Wang, Xuanqing Liu, Jiachen Li, Xin Lan, Yueqian Lin, Wengao Ye, Junwei He, Songlin Li, Yue Zhang, Yipeng Gao, Yijiang Li, Ze Ma, Liqiang Jing, Tianyu Wang, Kaixin Li, Yiqi Xue, Haoran Lyu, Yizhuo He, Yuchen Tian, Shutong Wu, Bowei Wang, Yixuan Gao, Bo Chen, Litong Liu, Sikai Cheng, Jiajun Bao, Shuaicheng Tong, Shuwen Xu, Terry Yue Zhuo, Tinghan Ye, Qi Qi, Miao Li, Longtai Liao, Zelin Tan, Chang Shi, Xilin Tang, Srinath Tankasala, Boqin Yuan, Yaoyao Qian, Jianhong Tu, Chenguang Wang, Yizhou Sun, Wei Wang, Aaron Taylor, Ziyue Yang, Changkun Guan, Zhikang Dong, Xinyu Zhang, Steven Dillmann, Han-chung Lee, Dawn Song
cs.AI
2026-02-13
SkillsBench's 87 tasks across 18 stacks: curated Skills lift pass rate from 33.9% to 50.5% (+16.6 pp); 1-3 focused modules beat large bundles, self-written Skills drop 8-11 pp.
Agent Skills are now a default layer in terminal agents: a folder with SKILL.md plus optional scripts, templates, and reference files, discovered and injected by the harness at inference. A construction snapshot counted 2,014,000 source-partitioned Skills. What was missing is a measurement: for this task, on this model-harness stack, does adding the Skill raise the pass rate, and by how much.
Existing agent benchmarks fold the model, the harness, and any augmentation into one number. SWE-bench and Terminal-Bench ask whether an agent can finish the job. They do not answer the deployment question: will this Skill help the same stack, and by how much. SkillsBench treats the Skill as a first-class evaluation object and isolates its effect with paired runs.
Tasks came from 400 submissions by 142 contributors and were filtered down to 87 items across 8 domains. Each task is a pinned container: a human-written instruction, a Docker environment with data, an oracle that must pass the verifier, and a deterministic pytest battery. Skills have to encode procedures for a class of problems, never the answer to one instance. Task instructions never name which Skill to load; the agent discovers them through ordinary progressive disclosure.
Evaluation runs on BenchFlow across 18 model-harness configurations. Every task is scored under two matched conditions: instruction only, and the curated Skill directory mounted. Three trials per cell. The headline metric is task-macro pass rate, plus normalized gain, the fraction of remaining headroom the Skill closes. A diagnostic third arm asks the agent to author packs with Anthropic's skill-creator, then solve using only those packs.
All 18 configurations improve. Mean pass rate moves from 33.9% to 50.5%, a +16.6 point lift and 25.5% normalized gain. The spread is wide.
| Stack | No Skill | Curated | Δ |
| OpenHands + GPT-5.5 | 51.5 | 67.3 | +15.8 |
| Codex + GPT-5.5 | 46.8 | 66.5 | +19.7 |
| Claude Code + Opus 4.7 | 43.0 | 61.2 | +18.2 |
| Gemini CLI + Gemini 3.1 Pro | 36.0 | 60.8 | +24.8 |
| OpenHands + GLM 5.1 | 32.7 | 58.4 | +25.7 |
| OpenHands + Gemini 3.1 Flash Lite | 16.0 | 20.1 | +4.1 |
| 18-stack mean | 33.9 | 50.5 | +16.6 |
The strongest absolute systems are not the biggest beneficiaries. GLM 5.1, Gemini 3.1 Pro, and DeepSeek V4 Pro pick up more than already-strong Opus 4.8 (+8.4) and Gemini 3.5 Flash (+7.1). Harness choice moves the same model by close to 10 points: Gemini 3.1 Pro reaches 60.8% in Gemini CLI and 52.8% in OpenHands; Opus 4.7 reaches 61.2% in Claude Code and 53.1% in OpenHands.
Self-authored Skills fall below the no-Skill baseline on all three dedicated-harness stacks: Claude Code −8.1, Codex −11.3, Gemini CLI −11.5. Trajectory audit points to packs the solver never opens, authoring time that displaces solving, and packs that lock in a wrong pipeline.
Domain gaps are large: Natural Science +28.8, Media & Content Production +24.1, Cybersecurity +18.9; Software Engineering +11.6, Mathematics & OR +9.7. The biggest single-task jumps start near zero: llm-prefix-cache-replay from 1.9% to 94.4%, dapt-intrusion-detection from 0 to 81.5%. On 13 of 87 tasks the curated Skill hurts. Typical failure: a heavyweight "correct" pipeline that crowds out a simpler default, or a solver the agent cannot debug.
Design ablations are more actionable. One Skill: +18.0 pp. Two or three: +19.0. Four or more: +10.1. Compact and standard-length docs (+19.0 / +21.5) beat detailed (+14.5) and comprehensive prose (+0.7). MiniMax M2.7 with Skills at 34.9% already clears GLM 5.1 without Skills at 32.7%.
Invocation is not uniform. Codex + GPT-5.5 reads a task Skill on 99.2% of curated trials; OpenHands + Gemini 3.1 Flash Lite does so on 46.4%. OpenHands + Gemini 3.1 Pro invokes 54.0% of the time, and pass rate conditional on invocation is 68.7%. High invocation does not guarantee a pass; low invocation is real on some stacks.
Three practical takeaways for anyone shipping Skills. Short, curated packs with executable scripts and verifier-facing format constraints beat encyclopedias. The same Skill can move nearly 10 points across harnesses, so evaluation has to include the real CLI. Letting the model write Skills on the spot and then solve with them is worse than using none, at least in this protocol. Small models with good Skills can match larger models running bare. That is a capacity patch, not a new backbone.
Every task is a terminal container. GUI agents, multi-agent setups, and very long horizons are out of scope. Adding a Skill also lengthens context, so some of the lift may be extra tokens rather than procedure; the self-authored arm mixes content quality with discovery failure and is not a clean length control. Benchmark Skills score 10.1/12; the ecosystem mean is 6.2/12, so dirty real-world packs will look worse. Negative deltas on 13 tasks show that a frozen "correct" pipeline with no cheap fallback can hurt. The paper itself flags the missing controls: random text, irrelevant docs, and retrieval-only baselines.