WorkflowArena: Benchmarking Executable Workflow Orchestration in LLM-Based Multi-Agent Systems
Zheng Jiang, Fang Niu, Bo Peng, Houde Qian, Wenyi Li, Deyao Hong, Mingju Gao, Yizhe Chi, Youjie Zheng, Kaisen Yang, Calvin Xiao, Lifeng Sun, Huan-ang Gao
cs.CL, cs.SE
2026-11-10
WorkflowArena benchmarks multi-agent orchestration on 145 long-horizon tasks: best config resolves only 36.95%; Opus-authored workflows lift weak executors by 6.41 points.
Agent harnesses like Claude Code and Codex now let a model spawn worker sub-agents at run time and write the executable workflow that coordinates them. The structure of a multi-agent system, once fixed by developers, is now decided by the model itself, and how well it organizes the work becomes a property of the model. No existing evaluation measures this. End-to-end benchmarks score the agent as a single black box, so the division of labor stays invisible; multi-agent benchmarks prescribe the coordination structure and end up measuring compliance; workflow-generation benchmarks score plans offline, never against real artifacts. WorkflowArena formalizes the capability as workflow orchestration and evaluates it directly on professional work: how the model partitions a task, what context each worker receives, and how intermediate results are verified and integrated.
On the task side, 145 tasks span 8 domains (software engineering, operations, science, formal mathematics, finance, law, content, 3D production) and 17 task forms, built from open-source repositories, mathlib4, SEC 10-Q filings, MAUD agreements, open films, and other professional sources. Most are reconstructive: the target work is removed while surrounding context stays, and hidden tests never enter the agent-visible workspace. Deliverables must integrate interdependent components, and each verifier includes checks that only an integrated result can pass, such as reconciliation across a contract portfolio or regression tests in software.
The protocol is the interesting half. Each task gets a four-hour wall-clock budget shared by the main agent and all workers, so time spent organizing competes with the work itself. The agent has a full shell plus a Workflow tool that launches background workflows of up to 16 concurrent workers, which it can inspect, revise, or stop. Every worker is a fresh instance of the same model and configuration, starting from a clean context and sharing only the workspace, so deciding what each worker needs to know is itself evaluated behavior. Native sub-agent mechanisms are disabled; delegation is optional and always recorded. A hidden verifier scores 0 to 100, and a task counts as resolved only when it meets every acceptance criterion of its domain.
Nine models, 25 model-effort configurations, same 145 tasks:
| Configuration | Overall resolved | Note |
| Claude Opus 5.5 + Claude Code (max) | 36.95% | best overall; leads SWE, SCI, MATH |
| Claude Opus 5.5 + Claude Code (high) | 34.11% | $6.66 per task |
| GPT-6 Astra + Codex (xhigh) | 30.80% | $29.26 per task |
| GPT-6.1 Sol + Codex (low) | 27.59% | $3.61 per task |
| GLM-5.3 Flash + Claude Code (max) | 4.43% | weakest |
No configuration leads every domain. MATH is near saturation at 95.83%, the best SWE rate is 29.41%, LAW never exceeds 9.09%, and FIN never exceeds 18.75%. Effort moves the two families differently: Opus swings from 22.02% at low effort to 36.95% at max, while Astra stays within 27.34% to 30.80%, and Sol at low effort actually beats its medium and high settings.
Delegation splits into two stable styles. At high effort, Opus and Sonnet average 0.86 and 0.76 Workflow calls per task with 5.55 and 6.14 workers per workflow; Astra and Sol average 2.09 and 1.94 calls with just 1.72 and 1.69 workers, and about 99% of their workflows contain at most four workers. The Claude systems delegate rarely with large crews, the GPT systems constantly in small units, and the domain reshapes the choice: in LAW, Opus averages 14.29 workers per workflow against roughly 1.7 for the GPT systems.
Pricing reorders the comparison. Opus at high effort resolves 34.11% at $6.66 per task; Astra at xhigh resolves 30.80% at $29.26, scoring 3.31 points lower at 4.4 times the cost.
The transfer experiment carries the most information. On a 32-task hard subset, the executor is held fixed (Kimi, DeepSeek V4.1 Flash, GPT-6 Luna, Doubao) and only the workflow's author varies. Workflows written by Opus raise every executor, by 6.41 points on average in static mode; Kimi goes from 35.75 to 46.61 under Opus dynamic workflows. Astra is itself far stronger than any of these executors (27.34% versus at most 16.48%), yet its workflows average a 2.53-point loss in static mode and lower Kimi and DeepSeek in both modes. Being good at a task does not automatically make a model good at organizing others to do it.
For anyone building agent systems, this benchmark turns questions like "should we go multi-agent, and at what granularity" from folklore into measurement, with two findings you can act on: there is no universally best delegation style, and the domain reshapes it; and strong-model-authors, weak-model-executes is a real cost-saving pattern, worth +10.86 points for Kimi. It is a benchmark paper, though. No new method components; the value sits in task construction, evaluation protocol, and empirical findings, and it depends on adoption.
The authors list four: each configuration runs once per task, so close rankings carry no variance estimate; each model runs in a single harness, so Claude-versus-GPT differences confound model with harness; content and 3D deliverables are graded by Gemini-3.8-Flash against rubrics, with human agreement unmeasured; and the transfer experiment covers only 32 tasks and two source models, and since sources also write worker instructions, the gains may mix organization with task knowledge conveyed in those instructions. Two gaps the paper does not address: no sensitivity analysis for the uniform four-hour budget across domains, and the equal-weight Overall averages domain rates regardless of task counts, so single-domain noise flows straight into the headline number.