Signal or Noise? A Benchmark Study of Agent Skills in Web Development
Ziyue Yang, Fan Ding
cs.CL
2026-08-24
Injecting a matched public WebDev Skill drops mean Pass@2 by 1.3–4.2 pp, raises tokens 72–394%, and helps only 17–36% of pairs.
An Agent Skill is a reusable procedure pack: a SKILL.md file plus optional scripts, references, and examples. Once attached, the same text is injected into every turn of a coding-agent session so the model inherits framework conventions, anti-patterns, and tool usage. Public marketplaces now ship thousands of these packs, and many products attach them at session start by default.
Injection lengthens every prompt. Existing numbers disagree. SkillsBench reports a +16.2 percentage-point gain on mixed domains; SWE-Skills-Bench finds 39 of 49 Skills with zero pass-rate lift on SWE-bench-style work. Native WebDev suites such as Web-Bench and ArtifactsBench measure generation with the prompt held fixed. Nobody has asked the operational question on realistic web tasks: given a Skill that already matches the stack, should it have been injected at all.
Baidu NLP built WebDev-Skills-Bench on top of ByteDance's Web-Bench: 50 projects, 11 stack categories (React/Vue/Angular/Svelte, Express/Fastify, ORM, CSS, Canvas/SVG/Three.js, bundlers, DOM apps), 20 sequentially dependent tasks per project, 1,000 tasks in total, scored by deterministic Playwright tests rather than an LLM judge. The Skill suite is 31 third-party public packs from visible repos (Anthropic docs, Vercel Labs, Addy Osmani, and similar). None of them were written for this study. SKILL.md lengths span roughly 1.2K to 22K characters; that file is the only text that enters the prompt.
Two annotators with professional WebDev experience labeled all 1,550 (Skill, project) pairs as core (declared frameworks or domain cover the project's primary stack) or skip (no meaningful overlap). About five minutes per pair, roughly 258 person-hours. Raw agreement 96.5%, Cohen's κ ≈ 0.74. The 55 disagreements were resolved conservatively toward skip. That leaves 117 core pairs covering all 50 projects.
The harness freezes the workspace, task order, decoding, and Playwright suite. Only the prompt-level SKILL.md changes.
Auxiliary files are mounted under .skills/<id>/ rather than concatenated into the prompt, so a multi-file Skill still has a well-defined length control. The panel is Claude Sonnet 4, GPT-5.1, DeepSeek-V4-flash, and Qwen3-Coder-30B-A3B. Greedy decoding, temperature 0, 64k maxTokens, N=3 seeds per cell, workspace reset with git clean -fdx before every run.
Mean ΔPass@2 is negative on every model.
| Model | ΔPass@2 | ΔPass@1 | ΔTCD | Win/Tie/Loss | token overhead |
| Claude Sonnet 4 | −4.2 pp | −2.8 pp | −0.85 | 30/9/61 | +72% |
| GPT-5.1 | −1.3 pp | −1.6 pp | −0.26 | 35/20/45 | +74% |
| Qwen3-Coder-30B | −2.3 pp | −3.2 pp | −0.47 | 17/35/48 | +91% |
| DeepSeek-V4-flash | −2.0 pp | −4.1 pp | −0.40 | 36/16/48 | +394% |
Three of four Pass@2 intervals exclude zero. GPT-5.1's Pass@2 interval just includes zero; its Pass@1 interval does not. Task Completion Depth, the longest consecutive Pass@2 prefix on the 20-task chain, falls in parallel, so this is not a pass-rate artifact. DeepSeek's +394% is inflated by early C0 failures shrinking the denominator; the direction matches the rest of the panel: more tokens, shallower chains. A positive tail still exists. Win rates sit between 17% and 36%, worst on Qwen, highest on DeepSeek.
The loss concentrates on easy early tasks. Every model's easy-bucket interval excludes zero: GPT-5.1 −4.0 pp, Sonnet −5.3, DeepSeek −7.3, Qwen −10.7. Moderate and challenging buckets are noisier, with ceiling and floor effects; Qwen has too few cells to report; DeepSeek is +11.7 pp on moderate tasks. Skills hurt most where the model already has a strong prior.
The proposed mechanism is retry lock-in. Web-Bench gives two attempts. A first-try mistake in button text, class names, or nesting is cheap if the second try can vary those choices. A Skill that pins the choices in place turns recoverable misses into chain-ending failures. In the Sonnet × zustand × react-expert trace, C0 Pass@2 is 55% and C1 is 40%. Both first attempts fail Playwright strict mode because the heading and the submit button both say "Create Blog". C0 retries with "Submit" and the chain reaches task-11. C1 retries with "Create Blog Post" to follow the Skill's naming rule; the substring still collides, and the chain dies at task-4.
The length-matched control splits the average into two mechanisms. Sonnet and Qwen are length-distracted: Sonnet ΔLength = −3.3 pp, ΔContent = −0.9 (content CI includes zero); Qwen ΔLength = −3.5, ΔContent = +1.2. An equally long irrelevant Skill reproduces most of the loss. GPT-5.1 and DeepSeek are content-misled: length terms near zero (−0.2 and −0.6), content terms negative (−1.1 and −1.4). Extra tokens alone are roughly harmless; the matched text steers the model off target. Most C1 wins still survive the length control: Qwen 95%, GPT-5.1 71%, DeepSeek 64%, Sonnet 60%. Model-level tendencies do not certify any single pair.
Cross-model transfer is near zero. Pearson on the 117 pairs sits between −0.08 and +0.12; Spearman |ρs| ≤ 0.16. 74% of pairs have at least one positive and one negative model sign. Only 1% win on all four models, 4% lose on all four. On lowdb × database-optimizer, Sonnet goes 33% → 67% (+33 pp), DeepSeek 42% → 20% (−22), Qwen 22% → 0% (−22), GPT-5.1 22% → 20%. A 55-point swing on one core pair. Baseline difficulty does not explain it: DeepSeek has the strongest C0 and the steepest drop.
C3 runs only on five pairs with a stable cross-model gain, where the full Skill is +5.1 pp, opposite in sign to the panel average. The numbers describe how a helpful Skill is built. Anti-patterns are the only slice with a reliable task-level direction (McNemar 111 vs 74, p=0.008). Positive rules average 0.0 pp. Example code averages −0.7 pp, yet dropping it saves 34,482 input tokens per run, about 22.7% of the SKILL.md budget. Cheap "don't" rules are the best cost-to-signal ratio. Drop Sonnet and examples become the strongest positive term (+4.2 pp, Wilcoxon p=0.005): DeepSeek +8.3, Qwen +3.7, GPT-5.1 +0.7, Sonnet −15.3. On the strongest model, in-skill examples act as a constraint that suppresses better priors.
Unconditional injection at session start loses on this benchmark. Mean pass rates fall, tokens rise 72% to 394%, and winners are a minority. A Skill is a hypothesis about a (Skill, project, model) triple, not a portable asset. A single marketplace ranking by stars will not survive a backend swap. Validating on Sonnet and shipping to a cheaper model is unsafe.
Three practical rules follow. Treat injection as opt-in. Route by chain position, not only by stack: skip early easy tasks, attach a Skill once error rates rise. Require a length-matched irrelevant control in any Agent-Skill benchmark, or length distraction and content misalignment collapse into one "Skills don't help" headline.
Writers of Skills should keep anti-pattern rules short and keep example dumps off the default path. Examples help weaker models, hurt Sonnet, and dominate token cost. In the helpful minority, the part worth keeping is the prohibition, not the sample.
Across three Sonnet seeds, aggregate C0 and C1 Pass@2 move by 4.4 and 3.6 pp, comparable to the headline effect. Model-level means are stabilized by N=3; pair-level estimates need a discount. C2 used 109 unique length-matched runs for 117 pairs because some prompts were shared within a project; a cluster bootstrap and full de-duplication are still open. Routing is conservative: only core pairs are scored, so off-target deployment is unmeasured. All Skills come from high-visibility public repos; enterprise packs or router-tuned Skills may behave differently. This is a pre-deployment audit, not an online A/B test: no live traffic, no human intervention, no product acceptance criteria. Metrics are Playwright functional correctness and token overhead. Readability, accessibility, visual fidelity, and review time are off the books.
C3 covers five already-helpful pairs and cannot be read as the average Skill. The easy-bucket loss is clear; later-chain cells are sparse, so "Skills help on hard tasks" is not established. Web-Bench's two-attempt budget is part of the retry lock-in story; a single-attempt protocol or looser locators could change the strength of that mechanism. The suite measures Skills as they exist on public GitHub, not an upper bound on a carefully written internal pack.