Agent Seer: Synthesizing Scenarios from Specification Understanding
Harish Karumuri, Mahesh Vemula, David Lopes Pegna
cs.CL
2026-06-25
From MCP specs only, Agent Seer builds eval harnesses on seven suites: mean tool-calling 0.911, full coverage on small/medium specs, with argument values as the dominant failure.
Tool-using agents are already wired into calendars, project trackers, and internal stores. Evaluation has not kept up. New APIs, private MCP servers, and suites that change weekly have no ready-made tests. Hand-written benchmarks cannot cover the combinatorial tool graph. A frozen suite ages as soon as the production schema moves. Multi-turn tests are worse: follow-ups have to react to actual tool returns, which you do not have if the tools are new or private. GAIA, AgentBench, and MCPVerse either need curators or live calls. None of them answers the cold-start case: the suite just shipped and there is no exam.
Agent Seer's claim is that the spec already carries enough semantics. Function names, natural-language descriptions, and typed parameter schemas let an LLM invent plausible workflows, fill arguments, fake tool responses, and grow those into multi-turn dialogues. The bottleneck moves from hiring a domain expert to extracting structure.
Four stages, each consuming schema-validated JSON from the last, so a malformed artifact dies at the boundary instead of leaking downstream.
The artifact is a self-contained harness: prompt, expected tool sequence, mocks, conversation, and a held-out oracle. A downstream MCP-compatible runner feeds mocks and scores against the oracle with no live backend. Generation uses Gemini 2.5 Flash Lite at temperature 0.7, with up to three structured-output retries before a record is discarded.
Seven public MCP specs, from Illustrator (64 tools) down to Filesystem (14). 337 scenarios, 391 evaluation records. The primary judge is Gemini 2.5 Flash at temperature 0.
Mean unsupervised tool-calling (TC) is 0.911 (95% bootstrap CI [0.897, 0.925], median 0.979). Mean coherence is 0.855 (CI [0.838, 0.872]). 31.7% of records score a perfect TC; 2.3% fall below 0.5. Coverage is 100% on the six specs with 14-56 tools. Illustrator, at 64 tools, drops to 56%, the only coverage ceiling in the experiment.
| MCP | tools | mean params | TC | coherence |
| Redis | 47 | 2.1 | 0.966 | 0.902 |
| Selenium | 56 | 1.8 | 0.935 | 0.850 |
| Elasticsearch | 20 | 1.8 | 0.930 | 0.902 |
| Illustrator | 64 | 3.6 | 0.898 | 0.855 |
| Slack | 16 | 2.2 | 0.886 | 0.938 |
| Filesystem | 14 | 1.8 | 0.876 | 0.825 |
| Git | 33 | 11.2 | 0.857 | 0.757 |
Quality tracks parameter-schema complexity more than suite size. At the MCP grain, mean parameters per tool correlates with TC at r=-0.60, optional-parameter fraction at r=-0.66; tool count is only r=+0.40. Git has 33 tools but 11.2 parameters on average (95% optional) and the lowest TC at 0.857. Selenium has 56 tools, flat parameters, and 0.935. At the tool grain (n=222), parameter count and optional fraction still correlate negatively with TC (r=-0.29 / -0.30, p<0.001). Complex scenarios drop 7.3pp TC and 5.3pp coherence versus simple ones; Elasticsearch loses 12.2pp, Git 11.1pp, Selenium barely moves.
Failures only show up once you split arguments. Usage is near-perfect (98% full credit). Selection is fully correct on 77% of records. Ordering zeros out on 10% of multi-tool scenarios. Arguments are fully correct on 42% and partial on 57%. Assigned to the lowest sub-dimension, value accuracy owns 223 records, then relevancy 44, format 35, type 31, completeness 16, name 11. Redis's typical miss is omitting an optional expiry on set while name, key, and value are right; a name-match metric would mark those as perfect. Git also leaks pretraining: three scenarios emit fetch, revert, and filter-repo, which are real Git CLI verbs but not in the MCP spec. That is 3 of 893 calls in the whole corpus, all in Git.
An out-of-family re-score with Qwen3.5-122B-A10B-FP8 keeps TC: paired r=0.79, MCP rank Spearman ρ=0.86, no mean shift. Coherence moves: about 0.16 lower, paired r=0.42, and the worst-coherence MCP swaps between judges. Treat TC as stable and coherence levels as judge-dependent.
Private or freshly shipped MCP suites need an exam more than they need another public leaderboard. This pipeline can turn a spec into a runnable regression pack without standing up the real service. The useful setting is a medium-size, relatively flat schema that you want to regenerate whenever the API moves. It is a cold-start tool. It does not replace a hand-authored benchmark.
Name match is the wrong passing bar. Argument values are the main remaining failure, so Git-like suites (dense optionals, the same parameter name with shifting meaning) need a human to check the args. Complex scenarios lose another 7.3pp, and multi-turn expansion succeeds only 16% of the time. Do not expect τ-bench-length dialogues out of the box.
The authors put the largest risk first: the oracle is LLM-generated, so systematic bias follows the generator. The framework is a proxy for relative gaps, not a gold label. Mock outputs for chained calls are generated independently, so IDs and values need not line up across a workflow. Coverage falls to 56% at 64 tools, with no strategy tested beyond that. Simple versus complex is prompt framing, not a structural filter. Seven MCPs, one generator (Gemini 2.5 Flash Lite), and only 54 multi-turn records cannot support general claims.
Generator and judge are both LLMs. A stronger Gemini 2.5 Flash scoring a weaker Flash Lite, plus the Qwen 3.5 replication, keeps the TC ranking honest. There is still no human study. All 871 mocks sit at low grounding, so response fidelity has no external anchor. Reading 0.911 as "these scenarios already look like real workflows" overclaims the evidence.