Agent Teams Beat Solo by Up to ~10 Points but Cost 4–7x More
nouhadziri · x · 2026-10-14
Evaluating agent teams from the GPT (5.6 Sol, Terra, Luna), Claude (Opus 4.8/4.7/4.5), and Gemini 3.7 families on ProgramBench: teams beat solo agents by up to 10 points, but at 4–7× the inference cost.
ProgramBench is reverse engineering: the agent gets only a binary and docs, must run it, infer behavior, and rebuild programs like jq, SQLite, and FFmpeg from scratch — the best agent passes only 10% of FFmpeg tests.
More from coding & agent
- 7 lessons from building a personalized AI gift site in one hour, with $0 hosting — jamesbrooksco · 2026-10-15
- Google's ADK brings one agent framework to Python, Go, TypeScript, Java and Kotlin — rseroter · 2026-10-15
- AI agent flags a bizarre '71 >= 71' error while a server pool balloons by 1.4TB — haydendevs · 2026-10-15
- Invideo launches an MCP for editing videos directly via Claude or ChatGPT — azed_ai · 2026-10-15
- Connecting Grok to ROS via OpenClaw and AgenticROS for robotics missions — chrismatthieu · 2026-10-15
- Automating bug report fixes: a Bot triages DMs into a Cursor project — msfeldstein · 2026-10-15