Dropping the vector DB from agent tool selection: same recall, cost basically gone

BenefitGrand8752 · reddit · 2026-09-18

A developer running a self-hosted personal assistant (local Gemma planner, 96 tools, Telegram + HTTP) removed the embedding model from tool selection entirely — recall didn't move, and cost basically disappeared.

The approach

Old: embed tool descriptions + query (BGE-M3, ONNX), nearest-neighbor top-k
New: token overlap over tool names plus four typed rules; pure CPU, 17ms, no model loaded
Key insight: tool names follow a closed compositional grammar (verbobject[qualifier], 23 verbs × 21 objects, e.g. readfilespdf), so query and tool tokens share a small lexicon and plain overlap is already strong — embedding becomes redundant
Rules: path/extension boosts, query-pattern boosts, verb→producer-family compatibility, rare-token-unmatched penalty

Numbers

Benchmark: 96 real tools, 234 organic queries, frozen corpus, deterministic, ground truth = actual production calls
Production config: Recall@5 0.786, Recall@1 0.487, 10ms mean; token-flat tied the dense baseline at every catalog size

Context

Lexical beating dense is already documented: BM25 NDCG@5 0.853 vs 0.834 dense on ToolBench (arXiv:2502.15526); tool-name fragility in arXiv:2505.18135; best practice remains hybrid retrieval
The novel framing: deliberately engineering a closed vocabulary so retrieval doesn't need to solve fuzzy semantics. Benchmark is public and reproducible.

A compelling 'less is more' datapoint for agent tool orchestration: if embeddings buy nothing, they just cost money on low-power boxes.

Original post →

Dropping the vector DB from agent tool selection: same recall, cost basically gone

More from coding & agent