Paper: Comparative Approaches to Agent Retrieval over Large Skill Libraries

alex_verem · x · 2026-08-10

This paper investigates how agents backed by large skill libraries decide which skills to load and in what order.

Experiment Setup: Tested a hybrid ranker (combining lexical and dense-embedding retrieval) against a typed knowledge graph over a corpus of 690 skills.
Performance: On 117 realistic queries, the hybrid ranker retrieved the correct skill in the top 5 with 73.5% accuracy, significantly outperforming the graph (-11.2 points).
Graph Limitations: The graph cannot extend retrieval reach because its candidate edges are drawn from the same embedding neighborhood the ranker already searches, with 98.6% overlap.
Evaluation Pitfall: Evaluating on author-written queries overstates hit@5 by up to 44 points, masking true system limitations.

More from coding & agent