RecToolBench: Benchmarking Recommendation-Specific Tool Orchestration under Fuzzy User Intent
Xiao Chen, Yicheng Zhao, Yingying Wu, Zhendong Chu, Changyi Ma, Qingsong Wen, Xuan Song
EMNLP 2026
cs.IR
2026-09-25
RecToolBench's 1,288 executable fuzzy-recommendation tasks show proprietary models above 95% Hit Ratio on serial chains, then 54% on hybrid L4 orchestration.
Recommenders are turning from passive rankers into instruction-following agents that call external tools. Existing benchmarks either assume a fully specified user intent or test isolated function calls. Real requests are fuzzy: "around four stars, budget thirty or forty, maybe a hot-shoe cover." The agent has to fill in parameters, chain tools across servers, and fold the evidence into a grounded recommendation.
A syntactically valid tool call is not a successful recommendation. That gap had no executable benchmark.
RecToolBench sits on the Model Context Protocol (MCP), a shared schema for tool calls. It covers Amazon Electronics, Yelp, and Foursquare with 13 MCP servers and 32 tools for retrieval, review sentiment, rating filters, personalized recommendation, geo reasoning, and web search. Tasks are built with a synthesize–fuzzify–judge pipeline: feasible workflows are generated from real interaction sequences and tool schemas, then explicit instructions are blurred with five ambiguity types (underspecification, soft contradiction, vague quantifiers, preference uncertainty, implicit constraints), then filtered and human-checked.
Difficulty has four levels: L1 single tool, L2 parallel, L3 serial chains, L4 hybrid parallel-plus-serial. The set has 1,288 tasks (300 L1, 988 L2–L4; 675 / 465 / 148 by domain). Agents run a multi-round plan–execute loop. Rule metrics check schema, execution, tool selection, and Hit Ratio. An LLM judge (DeepSeek-V3.2) scores parameter appropriateness, dependency awareness, and grounding. L4 has no single gold workflow, because several orchestrations can be valid.
On L1, proprietary models reach 100% tool-selection accuracy with near-perfect schema and execution. The strongest small model, Qwen-2.5-7B, selects the right tool 58% of the time and gets parameters right 45%. Gemma-3-4B has a 99% valid tool-name rate and 49% selection accuracy; Qwen-2.5-3B executes 95% of calls and lands parameters 10% of the time. Format compliance is cheap; semantic grounding is not.
On L2, small models look competitive on schema, then lose on completion. Hit Ratio: Gemini-3.1-Flash 55.59%, Claude-Sonnet-4.5 47.29%, GPT-5.4 43.78%, Qwen-2.5-7B 31.58%, Llama-3.2-3B 17.54%. Proprietary models add auxiliary tools, which lowers Tool Selection F1 and raises hits.
L3 splits execution from completion. Llama-3.1-8B posts 97.01% schema compliance and 92.15% execution success, with only 59.30% Hit Ratio. GPT-5.4 and Claude reach 96.67% and 95.00% Hit Ratio, with stronger dependency and grounding scores. Qwen-2.5-7B leads small models on Tool Selection F1 (60.18%) and still hits 52.76%.
L4 is the bottleneck. GPT-5.4 Hit Ratio is 54.05%, Claude 44.59%, Gemini-3.1-Flash-Lite 39.19%, small models 20%–28%. In GPT-5.4's L4 traces, dependency-reasoning failures are 54.1% and target-retrieval misses 53.6%. Llama-3.1-8B averages 68.2 calls per L4 task against a gold budget near 4.2.
For recommendation agents and MCP tool layers, "can it emit JSON" is the wrong pass/fail. Parameter grounding under fuzzy intent, cross-step dependencies, and evidence closing are where scores drop. Four difficulty rungs split failure modes: small models already miss tool choice at L1; frontier models still break dependency chains at L4.
The benchmark is a clean MCP sandbox. Industrial noise, latency, and live catalogs are not in it.
The environment does not match production latency or dirty data. Three domains and 32 tools are a thin slice of a real tool ecosystem. Agent designs were not fully swept, for cost. Tasks are synthesized with GLM-5.1 and then fuzzed, so gold workflows may echo the generator's tool habits. L4 has no reference plan, so Hit Ratio is strict when several recommendations are reasonable. The LLM judge is still a model grading a model, even with prompt shuffling.