Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools
Atul Anand, Sourav Chattaraj
cs.AI
2026-08-05
Six canary probe types turn a wrong-tool pick into a typed diagnosis; tool-selection safety doesn't track tier, and mid-tier GPT-4.1 trips more canaries than GPT-5.2.
Agent benchmarks tell you the agent failed, but not which reasoning step broke. End-to-end suites like AgentBench, ToolLLM, and τ-bench score task completion; distractor suites like MetaTool and MCPAgentBench drop irrelevant tools into the set and check whether the agent gets misled, but still return a single bit: it picked wrong. A developer fixing a tool-selection failure first needs to know what kind of mistake it was.
The authors borrow the misconception-probe idea from educational testing: instead of marking an answer right or wrong, design each item so a particular wrong answer maps to a particular misconception. They move this to tool selection as the canary tool, turning one wrong pick into a typed diagnosis.
A canary is a perturbation of a real tool along a single axis (name, description, parameter schema, or scope), left superficially plausible but wrong for the task. Six types, each probing one capability:
Plant the canary in the tool set and watch whether the agent picks it. The framework ships a schema-driven canary generator (works on any tool set), a realistic sandbox with 12 real tools across five MCP servers, a unified agent loop (hosted via LiteLLM, local via Ollama), a provider-independent judge (DeepSeek-V3.2 at temperature 0), and a trap detector that labels each call as real, canary, or unknown. Baseline CSR is 0 for every model, confirming canaries are the only source of trapped calls.
A worked example: the task "Convert 5 miles to kilometres" has a real tool convertunits and a canary advancedconvertunits labeled "optimized, research-grade". GPT-4.1 calls the canary, which returns a value flagged "may be unreliable", a capability mirage.
Eight models, 120 tasks (40 easy, 40 medium, 40 hard), three canary-density conditions times three seeds, for 8,640 task runs, plus a 2,880-run subtlety ablation.
| Model | Tier | CSR | Task success (TSR) |
| Opus 4.8 | frontier | 0.010 | 0.77 |
| Sonnet 4.6 | mid | 0.034 | 0.84 |
| Gemini 2.5 Flash | mid | 0.041 | 0.63 |
| Gemini 2.5 Pro | frontier | 0.049 | 0.61 |
| GPT-5.2 | frontier | 0.178 | 0.64 |
| Qwen3-8B | small | 0.291 | 0.45 |
| GPT-4.1 | mid | 0.311 | 0.61 |
| Llama 3.1 8B | small | 0.378 | 0.23 |
CSR spans about 36x from Opus to Llama (chi-squared = 1467, dof = 7, p < 0.001). Three findings: susceptibility drops sharply with capability; tier does not predict safety, since the most susceptible hosted model is mid-tier GPT-4.1 (0.311), worse than its frontier sibling GPT-5.2 (0.178), while within Google the cheaper Flash (0.041) beats Pro (0.049) and Anthropic's two models are the most resistant regardless of tier; and the taxonomy is capability-stratified, because capability mirage is the only probe that fires on strong models (hosted average 0.216; the other five sit near-inert at 0.008 to 0.094), whereas on the small models almost every type fires (Llama 0.106-0.255, Qwen 0.024-0.292).
Judge credibility: the primary judge DeepSeek-V3.2 and a second independent judge GLM-5 agree at Cohen's kappa = 0.75; a 40-run hand check reaches kappa = 0.90 (95% agreement, and every disagreement had the judge as the stricter side). CSR correlates with task failure (Spearman rho = -0.34). A subtlety ablation that softens each canary's give-away phrase leaves frontier CSR essentially unchanged (0.079 to 0.075), evidence the probes measure reasoning rather than phrase-spotting.
Three practical levers. First, do not assume a bigger model is safer at tool selection; probe each deployment directly. Second, description hygiene: avoid superlative capability claims ("research-grade", "more powerful") in MCP tool descriptions, since even frontier models over-trust them, and this is the one probe that keeps catching models as they get stronger. Third, the diagnostic is cheap to adopt, because layering a small panel of typed canaries onto an existing tool-use suite turns binary pass/fail into a reason code at near-zero cost.
Recovery is a separate, capability-aligned axis: a model that springs a canary but recovers loses little (Opus recovers from 0.82 of traps, Sonnet 0.68), so a cheap post-call check that notices a staleness or reliability flag can convert a trap into a near-miss. The weakest models fail on both axes: Llama traps most and recovers least (0.18).
Author-acknowledged: tool outputs are synthetic, so fact-specific tasks are scored on procedural coherence rather than factual correctness, lowering TSR uniformly; the small tier covers only two 8B models (Llama, Qwen); the suite is 120 single-authored, templated tasks, modest in size and positioned as a seed for an extensible framework (statistics rest on 8,640 runs and hundreds of per-type trap opportunities, not 120 raw points); the six hosted models come from three providers, one frontier plus one mid each, so provider and tier are partially confounded; judge validation is a 40-run hand check that a larger blind study would strengthen; and the subtlety ablation cannot fully separate a canary's tell from its lure, because for capability mirages the boast is both.
One caveat: at n = 8 models, "tier does not predict safety" is an observation, not a law. The within-provider inversions point both ways (Gemini's cheaper model is safer, but GPT's frontier is safer than its mid-tier), so the honest claim is that tier fails to order resistance, not that capability is irrelevant. The diagnostic framework holds up regardless of this conclusion.