AgentSWE benchmark: no coding agent comes close to delivering the agents it's asked to build
Jiahao Wang · hf · 2026-10-13
Researchers from BAAI, the Chinese Academy of Sciences and Renmin University introduce AgentSWE, the first benchmark testing whether coding agents can actually deliver the agents they're commissioned to build. Paper and code are open-sourced.
- Covers three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target.
- 25 tasks framed as commissions rather than exams: builders iterate on a handful of illustrative cases, while acceptance runs the frozen submission on held-out cases graded against requirement-derived rubrics with hard pass/fail gates.
- Evaluating a range of coding agents across models and harnesses, the authors find none comes close to delivering what the requirement asks for — the gap between what's demonstrated and what's delivered is the norm.
- Failure trajectory analysis identifies recurring modes: overfitting to visible cases, hollow runtimes behind seemingly complete code, and unsupported claims of completion — signals agents can learn from.
More from coding & agent
- React dev poteto: writing is thinking, so I never let agents write for me — andfanilo · 2026-10-13
- 7 types of AI evaluations every engineer should run, from functional to regression — goyalshaliniuk · 2026-10-13
- Claude Opus agents dig up buried public LIDAR data to model USS Nimitz to near-exact dimensions — teortaxesTex · 2026-10-13
- AGI Fleet launches: Linux and Apple silicon Macs execute and review AI work — Ghost_Pilot_MD · 2026-10-13
- UCSD's SPLATIFY Turns 3DGS Papers into Trainable Code, Cutting Weeks of Reimplementation to Minutes — UCSanDiego · 2026-10-13
- High agent-readability scores don't stop coding agents from botching API calls — Aliza_Solomon_DX · 2026-10-13