Towards a Science of Scaling Agent Systems
Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A. Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, Mark Malhotra, Paul Pu Liang, Hae Won Park, Yuzhe Yang, Xuhai Xu, Yilun Du, Shwetak Patel, Tim Althoff, Daniel McDuff, Xin Liu
cs.AI
2025-12-09
Fits a scaling model to 260 agent configurations (CV R²=0.373); multi-agent ranges from +80.8% to -70.0% vs single-agent, so architecture-task fit decides success.
"Add more agents" is many teams' default lever for LLM systems, but there has been no quantitative answer to when multi-agent helps and when it hurts. Prior evaluations mixed different prompts, tools, and compute across architectures, conflating architectural effects with implementation choices. "More agents is all you need" ignores that coordination itself carries a cost. This paper builds a predictive scaling model for agent systems, turning architecture choice from a gut call into something computable.
The authors run 260 configurations across six agentic benchmarks, five canonical architectures (single-agent SAS, plus four multi-agent: Independent, Centralized, Decentralized, Hybrid), and three LLM families (OpenAI, Google, Anthropic). The key design choice is standardizing tools, prompts, and compute so only the architecture varies, isolating its effect. They fit a mixed-effects regression with 20 parameters in four groups: base capability (Intelligence Index I), system config (agent count), task properties (tool count, single-agent baseline), and empirical coordination metrics (efficiency, overhead, error amplification, message density, redundancy). Quadratic and interaction terms capture nonlinearity such as "stronger base models gain less from coordination".
Cross-validated R² is 0.373, rising to 0.413 with a task-grounded capability metric. The model picks the best architecture for 87% of held-out configurations. Relative to the single-agent baseline, multi-agent performance ranges from +80.8% to -70.0%:
| benchmark | best MAS | change | worst MAS | change |
| Finance-Agent | Centralized | +80.8% | - | - |
| BrowseComp-Plus | Decentralized | +9.2% | Independent | -35% |
| PlanCraft | Hybrid | -39.1% | Independent | -70.0% |
| SWE-bench Verified | Hybrid | -2.1% | Independent | -14.9% |
Coordination overhead reaches 515% for Hybrid; Independent amplifies errors 17.2× while Centralized contains it to 4.4×. A fitted capability-saturation threshold sits around single-agent baseline 0.45, above which adding agents starts to lose.
Three patterns: coordination yields diminishing or negative returns once the single-agent baseline exceeds about 45%; tool-heavy tasks pay a multi-agent tax because the token budget gets fragmented; architectures without centralized verification propagate errors.
The practitioner takeaway is to stop defaulting to multi-agent. The paper converts "does multi-agent help?" from a belief into something you can estimate from baseline strength and task structure: decomposable tasks where the single agent is still weak (financial reasoning) gain +80%, while long-chain planning or tool-heavy tasks can lose most of the performance. The R²=0.37 is honest: enough to pick architectures, far from precise enough to predict absolute scores.
The authors flag several: SWE-bench and Terminal-Bench use only 20-instance subsets due to Docker cost; only four MAS configurations are covered, not an exhaustive sweep; the Intelligence Index correlates with the task-grounded ACI metric at only r=0.45. Most important, "agentic" is defined relative to current model capability, so today's agentic task may stop being agentic as models improve.