SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale
Guanqun Yang, Wenlong Zhang, Tian Shi, Ping Wang
cs.IR, cs.AI, cs.MA
2026-09-30
A training-free bi-encoder + cross-encoder retriever matches an LLM-mediated skill-retrieval loop on SkillsBench's 89 tasks, cutting per-trial cost from $51.30 to $27.54.
Anthropic's Agent Skills package procedural know-how into SKILL.md directories, and open-source aggregators have crawled past 238,000 of them. Selection, not authoring, is now the bottleneck. The obvious fixes both fail: loading the whole pool into context leaves pass rate at the no-skill baseline even at 192 skills while inflating token cost by 59%, and the lift is non-monotonic, with one relevant skill adding +17.8% but four or more regressing to +5.9%. The standing answer (Liu et al.) outsources selection to the agent itself: hybrid BM25 plus dense retrieval, then an in-loop process where the agent rewrites queries, inspects candidates, and synthesizes a task-specific skill. That synthesis is genuinely useful when no single skill suffices, but every task pays LLM tokens for the loop.
SkillSeek is a deterministic two-stage retriever with zero LLM calls, built from off-the-shelf parts:
The indexing text is the subtle part. Names and descriptions often miss the file-type surface forms the cross-encoder relies on, while appending the raw body lets implementation detail dilute the signal. Ablations: Tool-REX v3 beats plain name+description by 2.2%, and adding the body costs another 3.2%.
A 4x11 grid: two pools (192 curated, 34K marketplace), two backbones (Qwen3.5-397B-A17B, MiniMax-M2.7), 11 methods, 89 SkillsBench tasks under OpenHands, each trial capped at 30 turns and 600 seconds with graded pytest rewards in [0,1].
| Method | 192/Qwen3.5 | 192/MiniMax | 34K/Qwen3.5 | 34K/MiniMax |
| no skills | 0.352 | 0.267 | 0.352 | 0.267 |
| bm25 | 0.430 | 0.387 | 0.420 | 0.346 |
| SkillSeek (bge-reranker-v2-m3) | 0.480 | 0.371 | 0.409 | 0.346 |
| SkillSeek (Qwen3-Reranker-0.6B) | 0.430 | 0.353 | 0.442 | 0.341 |
| liuhybrid | 0.470 | 0.378 | 0.417 | 0.335 |
| liurefined | 0.397 | 0.344 | 0.442 | 0.329 |
Plain BM25 matches or beats liurefined on three of four settings; Qwen3-Reranker-0.6B covers the fourth at 0.442 vs 0.442.
The cost gap is larger (34K/Qwen3.5): liurefined totals $51.30 per trial, liuhybrid $39.43, SkillSeek $27.54, within fifty cents of the $27.41 no-skill floor. SkillSeek's retrieval runs on CPU at about 1.1 s median per lookup.
A first-stage recall ceiling explains the pattern. On the 192 pool BM25 already reaches R@5 = 0.546 and the reranker adds only 3.5%; on the 34K pool BM25 sits at 0.391, and the reranker adds 5.4%, growing to 9.9% with Tool-REX indexing. Reranker scaling plateaus near 0.6B (8B buys +0.7%), and open source beats the commercial APIs tested: Qwen3-Reranker-8B 0.450 vs Voyage rerank-2.5 at 0.435.
For anyone building agent systems, this is direct evidence to run the cheap baseline first: the standard IR recipe should be the default starting point for skill retrieval, and the savings are per-task tokens. The paper also contributes a diagnostic, the helpfulness gap (pass rate when the gold skill is retrieved minus pass rate on a miss). Voyage rerank-2.5 has the highest top-5 recall at 72.2% yet a negative gap of -0.025: retrieving its notion of the gold actively hurts the agent. The agent commits to rank-1 on roughly 65% of tasks, so whether the top-ranked skill is useful matters more than whether gold sits somewhere in top-5. LLM-mediated loops are best held in reserve for the case they are uniquely good at: synthesizing a new skill when no single pooled skill suffices.
The authors are candid. Each main-grid cell is a single trial; a repeated setting has standard deviation around 1%, so differences under 2% are noise and the claim is observed parity, not statistical equivalence. Everything comes from 89 tasks, one harness (OpenHands), two backbones, and two pools; terminal, web, and SWE agents are untested. Failed trials count as zero, depressing all absolute numbers. Two concerns about the comparison itself. liurefined is a reimplementation, and it flips sign against the original: Liu et al. report refinement helping by +4.1% on 192/Qwen3.5 where this paper measures -7.3%, so the strength of the baseline is uncertain. And the per-task analysis shows the largest losses on the 34K pool (-0.50 to -0.80) all come from ranking near-twin skills wrongly, a precision failure rather than a coverage failure, which is exactly where an LLM loop might genuinely win; the paper never tests that head-on.