Study on scaling coding agent teams: placement matters more than headcount

Researcher @nouhadziri released a study on "agent teams" as a way to scale test-time compute: teams can beat a solo agent by up to about 10 points but at 4–7x the cost. The core finding is that adding agents pays off only at a task's bottleneck—where you place the division of labor matters more than how many agents you use. Delegating only the planning stage, with a cheap model as the planner, retains most of the gains at far lower cost. The work provides systematic cost-performance trade-off data for multi-agent systems.

Confirmed

• Method: a minimal hierarchical lead-worker framework built with reference to mini-SWE-agent, where the lead dynamically spawns workers, assigns subtasks, and aggregates reports; workers see only their own subtasks and share a workspace. Controlled experiments study design choices such as organizational structure, communication, and model mix
• Benchmark ProgramBench: agents receive only a program binary and documentation with no source code, and must run the program, infer its behavior, and rebuild real software like jq, SQLite, and FFmpeg from scratch; tasks are extremely hard—even the strongest leaderboard agents struggle on FFmpeg
• Overall performance: across frontier models including GPT (5.6 Sol/Terra/Luna), Claude (Opus 4.8/4.7/4.5), and Gemini 3.7, teams beat solo by up to 10 points but cost 4–7x more
• Communication patterns: even though workers can privately message any agent, 84% of messages still flow to the lead (interim findings, interface proposals, test failures, questions); workers send almost nothing to each other
• Model differences: GPT-5.6 and Claude Opus 4.8 proactively delegate and benefit; Claude Opus 4.7 barely delegates; Claude Haiku 4.5 and Gemini 3.7 perform worse
• Heterogeneous teams: with cheap workers fixed as Luna and upgrading the lead (Luna→Terra→Sol), reward rises from 42.5% to 51.6% then 60.5%, beating a solo agent using the same model as the lead at every tier
• Scaling: manually giving a Sol lead 2/4/8/16 cheap Luna workers shows planning-stage reward growing steadily with headcount; 16 cheap planners approach the all-Sol team's score at 70% cost savings
• Cost breakdown: a Sol lead + Luna planning workers retains about 47.5% of the all-strong team's gain for only 13.9% extra cost; cheap models as implementation workers perform poorly
• Division of labor: letting workers handle only exploration and planning (P mode) matches full-stage delegation (PIV mode) in performance, but P mode costs roughly half as much
• Asynchronous design: an async lead that keeps working and polls while workers run finishes tasks earlier than a blocking lead

Why it matters

• Multi-agent costs balloon easily; this study offers a clear cost-saving path: identify the task's bottleneck stage (here, planning) and deploy cheap workers only at that bottleneck
• The "placement beats headcount" conclusion is a strong corrective to the industry's habit of blindly stacking agents
• It reveals behavioral regularities in teams—differences in delegation propensity across models and centralized communication (84% upward reporting)—providing empirical grounding for multi-agent system design

2026-10-14 ~ 2026-10-14 · 14 related posts

• [source] Lessons from Scaling Coding Agent Teams: Frontier Performance at 30% of the Cost — nouhadziri · 2026-10-14
• A Controlled Study of Agent Teams: How Design Choices Shape Performance and Cost — nouhadziri · 2026-10-14
• Inside the Minimal Hierarchical Agent Harness: Lead, Workers, Shared Workspace — nouhadziri · 2026-10-14
• Async Leads Finish Sooner: How the Lead Waits Matters in Agent Teams — nouhadziri · 2026-10-14
• ProgramBench: Agents Reverse-Engineer FFmpeg-Class Software from Binaries — nouhadziri · 2026-10-14
• [source] Agent Teams Beat Solo by Up to ~10 Points but Cost 4–7x More — nouhadziri · 2026-10-14
• Agent Teams Get Expensive Fast: How to Cut Costs While Keeping Gains — nouhadziri · 2026-10-14
• Delegate Only Planning: Agent Team Matches Full Delegation at Half the Cost — nouhadziri · 2026-10-14
• Cheap Planners Keep Half the Gain for a Seventh of the Extra Cost — nouhadziri · 2026-10-14
• Heterogeneous agent teams: the lead sets the ceiling, cheap workers set the price — nouhadziri · 2026-10-14
• Sol lead + 16 cheap Luna planners nears all-Sol team results at 70% less cost — nouhadziri · 2026-10-14
• Agent team study: 84% of messages still go to the lead, workers rarely talk to each other — nouhadziri · 2026-10-14
• Not every model knows how to lead: GPT-5.6 and Opus 4.8 delegate well, smaller models don't — nouhadziri · 2026-10-14
• [source] Agent Teams: Where You Place Agents Beats How Many, Study Finds — nouhadziri · 2026-10-14