Agent Architecture Cost-Success Trade-offs: A Controlled Evaluation across Enterprise-Style SOPs
Subhrangshu Nandi, Hitesh Arora, Yunfei Bai, Udita Patel, Chirag Shah, Hannah Kwon, Wei Zhang, Weian Chen, Chloe Ruoqi Li, Alberto Sconfienza, Ayush Goyal, Yasmine Lee, Ming Xiong, Prathap Ramachandra
cs.AI
2026-11-07
Eight arms on 13 SOPs (2,124 tasks), model and tools fixed: single agent is cheapest at 67.4% TSR; only rebuilt orchestration wins significantly, at 75.6% and 4.7× tokens.
Enterprises run LLM agents on SOPs, standard operating procedures with long, branching, tool-heavy steps. The agent reads the procedure, fills arguments, carries state, and commits to a decision. Four designs compete: a function-calling single agent, a single agent that loads skills on demand, centralized orchestration, and a triage swarm whose specialists hand work to each other. A skill is a reusable procedure writeup, with steps and constraints included.
Existing studies test skills or coordination separately. None pins the model, the tasks, and the tools, then changes only the architecture on long enterprise procedures. Extra agents get treated as an upgrade. The token bill and the latency arrive first.
The testbed is SOP-Bench. Mock tools return fixed values, so every arm sees the same external outcomes. Differences come from the architecture and from the skill text. The model is claude-opus-4-8 on a pinned code revision. Macro TSR on the baseline moves by at most 2.3 points across Opus 4.6, 4.7, and 4.8. Macro TSR is the mean of per-SOP success rates. The study does not claim the ranking for other model families.
The shared matrix is 13 SOPs, 2,124 tasks per arm, and eight arms.
A 50-point rubric scores a skill as a procedure, across discoverability, instruction quality, contract clarity, disambiguation, and governance metadata. The high corpus averages 99.8%, the degraded corpus 67.6%. The score describes the writeup. It is not used to predict TSR. The low corpus is a bundled cut of the high one, with identity and tool dependencies left in place.
TSR is correct tasks divided by all tasks. Multi-field outputs must match on every field. Between-SOP standard errors are about 7 to 9 points, against about 1 point if tasks are pooled, so more tasks do not stabilize an architecture ranking. Pooled comparisons are paired with a per-SOP sign test. Open-tool skill guidance (OSG), where skills are advice and every tool stays visible, runs on only the two most tool-heavy SOPs and stays out of the main matrix.
SOP difficulty moves success more than architecture does. Macro standard errors run from 6.3 to 7.8 points and overlap across all eight arms. No arm separates from the baseline on a per-SOP sign test.
Cost does separate them. The baseline uses about 24,900 tokens and 24.0 seconds per task. Every coordinated arm spends 1.4x to 4.7x the tokens and 2.6x to 5.8x the latency. No arm is both cheaper and more accurate. Orchestration v1 and the swarm with no blueprint are strictly dominated by the baseline.
| Arm | Macro TSR | Tokens, latency |
| Single-agent baseline | 67.4% ± 7.7 | 24.9k, 24.0s |
| Orchestration v1 | 58.2% ± 7.7 | 1.4×, 2.6× |
| Swarm, descriptive | 70.4% ± 6.6 | 1.9×, 4.2× |
| Swarm, no blueprint | 65.4% ± 6.9 | 2.1×, 4.4× |
| CSE, low-quality skills | 59.8% ± 7.8 | 2.4×, 2.6× |
| Swarm, full | 70.5% ± 6.8 | 2.7×, 5.8× |
| CSE, high-quality skills | 73.1% ± 6.3 | 3.0×, 2.8× |
| Orchestration v2 | 75.6% ± 6.3 | 4.7×, 3.1× |
The baseline beats v1 on 9 of 13 SOPs, 6 of them at p < 0.05, and v1 has the lowest macro TSR. The losses sit on coupled, multi-field procedures: patient intake at 13.6% and know-your-business (KYB) at 23.3%. Each worker sees only a slice, nobody can fill a missing input, and the pipeline abstains with a placeholder. The line stops because every station is waiting on a form nobody holds.
v2 closes that gap. Patient intake rises to 98.5%, email intent from 75.8% to 96.2%, KYB from 23.3% to 60.0%. A random-effects model on the 13 per-SOP log-odds ratios gives an odds ratio of 1.72, p = 0.007. v2 is the only arm that significantly beats the baseline, 75.6% versus 67.4%, an 8.2 point gain, and it is also the most token-hungry arm, about triple v1. The same multi-agent family loses when workers may abstain and wins once they must commit and may be re-dispatched.
High-quality skills beat low-quality ones by 13.3 points (73.1% vs 59.8%), mostly on KYB (+45.6) and email intent (+39.2). The sign test is 7 wins, 4 losses, 2 ties, not significant. The degraded corpus, at 59.8%, falls below the bare baseline. On referral v2 the degraded skill drops the output contract, so near-correct decisions are filed under an invented key and scored wrong.
A swarm blueprint helps once, then only adds cost. Macro TSR goes from 65.4% with no blueprint to 70.4% descriptive and 70.5% full. Descriptive and full are indistinguishable (sign test p = 1.00), while full spends 42% more tokens and 39% more latency than descriptive. On referral-abuse v2, the bare swarm over-escalates 55 of 200 tasks. The full blueprint's rule that every stage must run also overrides a legitimate early stop: on KYB it walks past an incomplete-documentation gate and flips the case to escalate.
On simple SOPs the baseline is already at 71.8%. On complex SOPs it falls to 63.6%, while orchestration v2 holds 72.9% and high-quality CSE holds 71.1%. Low-quality skills fall to 49.4%. The largest Spearman correlation between an arm's edge and complexity is about +0.38. At n=13, significance needs an absolute correlation above 0.56.
Tool scoping depends on the workload. On KYB, with 8 tools, high-quality CSE reaches 85.6%, high-quality OSG 68.9%, and the baseline 38.9%. On video annotation, with 26 tools, high-quality OSG reaches 57.6%, about 20 points above high-quality CSE at 37.6%. Two SOPs, so this split is exploratory.
Against the baseline, the descriptive swarm gains 17.8 points on KYB at 2.0× tokens and 3.6× latency, and loses 0.7 points on dangerous goods at 2.3× tokens and 4.4× latency. The break-even line drawn in the paper is a sketch, not a fit.
Start with the single agent. Measure marginal TSR on the SOPs you actually run, then decide whether orchestration or skills are worth paying for. High-quality skills buy about 5.7 points at 3× tokens. A descriptive swarm buys 3.0 points at 1.9× tokens. Neither gain clears the baseline at a significant level. The one significant arm, orchestration v2, costs 4.7× tokens.
Before switching families, check whether workers abstain through placeholders. A plausible multi-agent build loses to the single agent. Giving every worker the full context and one re-dispatch makes it the only significant winner, and the most expensive arm in the study. On fraud, moderation, and intake-style procedures, success is too low to run unattended, and a bad skill can emit a broken schema while looking as if it finished.
The tools are synthetic deterministic mocks, without live latency, faults, or nondeterministic returns. Every arm uses claude-opus-4-8. Each task is run once. Arms span multiple runs and code revisions. Process metrics lack reference labels and are not reported. Each architecture is a single implementation, and the skill contrast is a bundled degradation. Complexity correlations are not significant, and the OSG ablation covers two SOPs.
The paper also disagrees with itself. The abstract, the methods, and Table 3 use 13 SOPs, with orchestration v2 at 75.6% versus 67.4%, odds ratio 1.72, p = 0.007, and cost at 1.4x to 4.7x tokens and 2.6x to 5.8x latency. The conclusion and the limitations section switch to 12 SOPs, 74.4% versus 65.6%, odds ratio 1.78, p = 0.008, and 1.5x to 4.8x tokens with 2.5x to 5.9x latency. The complexity split is 6 simple and 7 complex in the text, and 6 each in the table caption.
Multi-field scoring is not fully cleaned up. An output-contract bug truncated decisions into near-zero cells. After the fix, swarm scores rose on aircraft inspection, email intent, and patient intake. Orchestration's multi-field cells are still marked provisional. The jump from 13.6% to 98.5% on patient intake sits in that set. The v1-to-v2 reversal still needs a recompute under the same fix.