AgentSWE: Can Coding Agents Build the Agent You Actually Want?
Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu
cs.SE, cs.AI, cs.LG
2026-10-09
AgentSWE's 25 commissions ask coding agents to build agents. The best Creation mean is 19.1, and development scores of 90 or higher average 9.9 on held-out cases.
Ask Claude Code or Codex to build an agent, and the standing assumption is that the program which comes back will do the job described. Existing benchmarks miss that question. SWE-bench checks whether a patch matches a reference test suite. NL2Repo-Bench, Paper2Code, and PaperBench grade ordinary code, or a paper reproduction, against an answer known in advance. AgentBench scores agents someone else already built. AIE-Bench asks whether a meta-agent can raise a pinned scaffold's score inside a controlled edit boundary. What is left open: given a natural-language requirement, does a general coding agent deliver an agent that still meets the clauses on cases it never saw?
AgentSWE defines fidelity from observable behavior. A requirement splits into a goal, an interface, constraints, and resources. A hard gate checks that the agent launches, its outputs parse, and its edits stay on the allowed surface. Fail the gate and the score is zero. Pass it, and the score is how well unseen cases satisfy the requirement. Development score minus held-out score is the gap between the demo and the delivery.
The suite is 25 commissions. The builder receives the requirement, a starting artifact, and a few public development cases, and feedback arrives only on those cases. Harbor runs one session and then freezes the delivery. Held-out cases replay that freeze in an isolated runtime. Deliveries in a task family share one runtime model behind a benchmark gateway, so the score tracks prompts, orchestration, state, and output checks.
A run that fails to launch, times out, or emits nothing parseable scores zero before any partial credit. A judge then applies a requirement-derived Result rubric from 0 to 100, using artifacts, traces, and state. GPT-5.6 Sol judges Creation. DeepSeek-V4.1-Flash judges Editing. GPT-5.6 Sol also drafted the task materials; people revised them. The suite contains 150 development cases and 365 held-out cases.
The main comparison pairs the Codex harness with five models, one build per task.
| Builder | Creation | Editing | S |
| DeepSeek-V4-Pro | 19.1 | 31.6 | -4.0 |
| DeepSeek-V4-Flash | 15.1 | 14.0 | -7.1 |
| GPT-5.6 Sol | 13.7 | 23.9 | 4.7 |
| GPT-5.5 | 5.8 | 31.6 | 1.2 |
| Qwen3.6-35B-A3B | 2.3 | 0.0 | 0.9 |
Fifty Creation runs average 11.2, with a median of 3.4, and 20 of them score zero. DeepSeek-V4-Pro leads and still sits under 20 on 8 of 10 tasks. Only three runs clear 50, all on formal theorem proving or repository bug repair. The three builders that deliver on nearly every Editing task land between 20 and 35. The single 70 is GPT-5.5 on Codex execution residual. Qwen3.6 never submits an Editing candidate and often stops once it has announced a plan. On Optimization, no builder closes more than 5% of the remaining gap on average. Of 25 runs, 13 beat the starter, 9 lose to it, and 3 tie. Four of five builders improve BrowseComp; four of five regress on PinchBench.
Eleven Creation runs finish development at 90 or above and average 9.9 held-out. One hard-codes its two development cases. One Editing delivery scores 100 in development and 28.3 held-out. Selected Optimization candidates gain 13.1 on the development split and 1.8 on the held-out split.
Three failures recur. Builders fit the visible cases in 14 of 25 Optimization runs and in 13 of the 33 Editing runs that deliver. On PinchBench, DeepSeek-V4-Flash gains 8.1 in development and loses 15.2 held-out; DeepSeek-V4-Pro gains 59.4 and loses 15.1. Both route by keyword into playbooks for categories that showed up in development. In 8 of 10 DeepSeek-V4-Pro Editing runs, development feedback already names a broken clause that held-out cases still fail. Hollow runtimes are common: 157 of 270 traced Creation executions (58.1%) die at the validity gate, mostly on an error exit or a timeout. Of 33 Editing deliveries, 28 leave some new capability missing, uncalled, or unfinished. One OpenWiki delivery passes every gate and still leaves four of five required output categories empty. Completion claims are unreliable. Six of 42 final Creation reports and 3 of 19 Optimization reports assert success that their own traces, artifacts, and feedback contradict. No report that admits an incomplete job is contradicted. Delivered Editing agents overclaim in 13 of 33 runs.
Changing the setup moves scores and leaves the gap. The development-case and prompt trials use Codex with DeepSeek-V4.1-Flash. Going from zero to two development cases raises Result by 34.3 on PDF translation and by 29.3 on PPTX. Two further cases leave PDF at 33.2 and drop PPTX from 33.5 to 15.2. A prompt that prescribes the architecture drops PPTX from 29.6 to 7.4. Asking the builder to write extra cases and run them end to end moves PDF from 39.2 to 48.9, inside the spread of three repeats. Raising the per-case limit from 600 seconds to 1,500 seconds leaves ten Creation tasks at 70.3 on the last development score and 31.5 held-out, with development ahead on every task, and about a third of held-out executions still invalid. The harness matters as much as the model. On scientific PDF translation, GPT-5.5 scores 21.8 under Codex and 42.8 under OpenCode; DeepSeek-V4-Flash scores 5.8 under Claude Code and 30.5 under DeepSeek Harness. No harness leads for every model.
If acceptance uses only the cases the customer already showed the builder, the score is inflated. Those 11 Creation runs at 90 or above average 9.9 once the cases change. Reading the source does not catch a hollow runtime. Set aside Qwen3.6, which is lowest on both, and a static code score is 44.2 for deliveries with zero Result and 45.8 for the rest. Extra examples, a prescribed design, a self-test instruction, and a 2.5x time budget all move the number. Development stays ahead of held-out under every one of those conditions.
What transfers is the acceptance protocol. Freeze the submission, run clauses the requirement states and the examples skip, and check completion claims against the trace. Report the harness and the model together. AgentSWE-Lite builds eight tasks three times per pairing, with both the runtime model and the judge swapped to DeepSeek-V4.1-Flash, so its levels are not on the same scale as the main table. Repeats of one pairing already scatter: GPT-5.6 Sol on DeepTutor scores 32.8, 33.3, and 90.5.
The main experiment does not show a way to close the gap. The follow-up it names is harnesses that test against the requirement and verify completion claims, plus training that rewards behavior on unseen cases.
Each main-table pairing is built once. Repeats of the same task land at 0, 33.3, and 56.8 for DeepSeek-V4.1-Flash on repository repair, and at 32.8, 33.3, and 90.5 for GPT-5.6 Sol on DeepTutor. A ranking from a single run is unstable.
The judge is a model, and it overlaps the subjects. GPT-5.6 Sol judges Creation and is also a builder. DeepSeek-V4.1-Flash judges Editing. The best Creation mean, 19.1, belongs to DeepSeek-V4-Pro, so the leaderboard does not simply award the judge's own family. Partial credit is still a model's taste. The same GPT-5.6 Sol drafted the requirements, cases, and rubrics. People revised them. There is no ablation of that overlap.
The case-count, prompt, and timeout studies cover two Creation tasks, one builder, and three runs per cell. They do not license a claim about all 25 tasks. Six held-out cases are written to exercise requirement clauses, which is a designed sample, not a user distribution. Locking every delivery to one runtime model measures orchestration and says nothing about which backbone a user would want inside the agent. Optimization has five tasks, and the sign of the gain follows the task. Further evaluation limits are deferred to Section J and are not spelled out in the main text.