AgentSWE benchmark: coding agents score just 11.2/100 building the agents you actually want
_akhaliq · x · 2026-10-13
AgentSWE is a new benchmark testing whether coding agents can build, edit, and optimize the agent you actually want from a natural-language commission, spanning Creation, Editing, and Optimization across 25 tasks. It fills a gap: existing software benchmarks check patch correctness, and agent benchmarks check task outcomes, but nobody tests whether the built agent faithfully meets requirements.
Results are bleak: 50 Creation runs average 11.2/100, with 20 scoring zero. Per-model scores: DS-V4-Pro leads with 19.1 Creation and 31.6 Editing; DS-V4-Flash 15.1/14.0; GPT-5.6 Sol 13.7/23.9 but -4.0 on Optimization; GPT-5.5 and Qwen3.6-35B-A3B barely register at 1.2 and 0.2 on Creation. The benchmark also measures requirement fidelity, generalization, and failure analysis.
Related event: AgentSWE Benchmark: Coding Agents Fail to Build Agents(2 posts)→
More from coding & agent
- Garry Tan endorses Aside Browser as the fix for agents getting blocked on the web — garrytan · 2026-10-23
- MCP server turned rogue 30 days after approval, prompting open-source mcp-guard lockfile tool — ErgoniaWorks · 2026-10-23
- Test: Coding Agents Won't Use an MCP Tool Unless AGENTS.md Mentions It — colinmcnamara · 2026-10-23
- Open-Source Tennis Models Plus Claude Turn Video Into a World-Class Coach — tweetsatpreet · 2026-10-23
- awesome-ai-apps: 132 Free Open-Source AI Agent and RAG Projects on GitHub — Arindam_1729 · 2026-10-23
- Google's Agents CLI Adds Extension System, Ships First-Party LangChain Support — rseroter · 2026-10-23