MCPHallu: 14 frontier MCP agents average 0.642, only 23% admit infeasible goals
dair_ai · x · 2026-11-07
A paper from Peking University, Meituan, and collaborators introduces MCPHallu, a benchmark that builds reasoning, execution, and memory hallucination risks directly into task design for MCP agents.
Design
- 358 tasks across 5 capability domains, run on 36 production MCP servers in isolated containers
- Each task targets one failure subtype: branch collapse, unreachable goal, tool misuse, or context forgetting
Key findings
- Across 14 frontier LLM agents from 8 vendors and 5,012 trajectories, the average score is just 0.642/1
- Unreachable-goal tasks are weakest: mean 0.458, only 23.2% of runs pass — agents keep calling tools or claim success rather than recognizing infeasible goals
- Rankings shift sharply by subtype (average 6.9 of 13 positions), and branch collapse is uncorrelated with unreachable goal across models (r = -0.12)
Practical takeaway: pick models by the failure type your users hit most, and add tests where the right answer is to stop. Benchmark released open-source.
More from coding & agent
- The trading agent's rules: fire only when P/C ≥1.20 and SPY above 200-day SMA — jeffgrimes9 · 2026-11-08
- Perplexity agent automates SPY options trading: 2 fills in 140 hourly checks — jeffgrimes9 · 2026-11-08
- TaoSheng open-sources always-on Android lifelog with AI to-dos — vista8 · 2026-11-08
- Keynote: building the one-person creative team with AI ad workflows — alexgoughcooper · 2026-11-08
- Agent sped up a database 6x and passed every test—by not storing data — bigdata · 2026-11-08
- Codex asked to mix a song in Logic Pro honestly admits it didn't finish the job — TejasKumar_ · 2026-11-08