Vals AI: Agent Teams Cost 1.8-5.1x More, Only One Score Gain Was Significant
rseroter · x · 2026-10-13
Vals AI tested whether multi-agent teams pay off on its Vibe Code Bench, which asks models to build complete web apps from a product spec, deployed via Docker Compose and graded by browser agents clicking through workflows. The work splits into database, API, interface and deployment, so seam bugs show up directly in the score.
Setup
- Models: GPT 6 Sol and Claude Opus 5.5, 50 apps each.
- Conditions: single agent vs team, at medium and max reasoning effort — eight setups, each run once.
- In a team, the lead can hand parts to up to five subagents at once; subagents cannot spawn their own. Leads got one short instruction: split into independent parts, delegate each, then integrate and verify.
Key findings
- Teams cost 1.8x to 5.1x as much as single agents.
- Only one score difference was statistically significant: Sol's medium-effort team gained 7.3 points over its single-agent run.
- Whether a team is worth it depends on model, task and budget; higher reasoning effort is an alternative worth weighing.
More from coding & agent
- Use your agent.ai email to sign up a Google account so your agent gets its own identity — RachelVT42 · 2026-10-13
- Rox trains an outbound reward model on seller edits: 5x faster, 30x cheaper than LLM judges, 1.75x reply rate — andrew_ilyas · 2026-10-13
- Developer says EU compliance turns AI coding agents into a 'EU token tax' — kevinkern · 2026-10-13
- Live talk: MinerU-Popo paper on fixing OCR's cross-page context failures — HamelHusain · 2026-10-13
- Amp coding agent adds GitHub Webhooks: PR reviews and CI results flow back into the same thread — HamelHusain · 2026-10-13
- Agents must be lawful by design: safety should live in the inference engine — rseroter · 2026-10-13