Testing When AI Agents Should Call Tools Using NP-Hard Problems
Acceptable_Block_591 · reddit · 2026-07-15
The author is building a small testbed to evaluate when an LLM agent should delegate tasks to tools, rather than just testing if it can. The design uses the Hamiltonian path problem: easy to verify but scaling in difficulty with graph size, causing model reasoning to fail on larger graphs. The test compares three settings:
- LLM reasoning alone
- LLM choosing to call a classic solver tool
- Forced delegation to the solver
A deterministic verifier checks answers, logging accuracy, token costs, and tool selection frequency as difficulty rises. The author argues existing benchmarks mostly test "can it call tools correctly," rarely "should it call tools"—the latter being the real failure point in production. Feedback on design and metric blind spots is requested.
More from coding & agent
- A roundup of AI agents and MCP resources, including how to evaluate agents — _jaydeepkarale · 2026-07-21
- Anthropic shares a masterclass on how it builds AI agents — _jaydeepkarale · 2026-07-21
- Anthropic masterclass spotlights how to build and observe AI agents — _jaydeepkarale · 2026-07-21
- A beginner guide to AI agents points readers to a Stanford webinar — _jaydeepkarale · 2026-07-21
- A full course shows how to build and deploy an AI agent with OpenAI and LangChain — _jaydeepkarale · 2026-07-21
- A practical guide on how to evaluate AI agents — _jaydeepkarale · 2026-07-21