SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
Xinping Lei, Minghao Liu, Ruijie Zhu, Xiaohui Hu, Xingqiao Li, Ao Zhang, Wenhao Chai, Han Li, Tao Wang, Jingyuan Zhang, Enduo Zhao, Yunfeng Shi, Yiming Liang, Zetao Yang, Xingjian Wang, Yifan Yao, Xinkai Ma, Yingru Li, Tianhan Yang, Huan Zhou, Weiran Shi, Shen Yan, Jiaheng Liu, Stephen Huang, Eli Cheung, Weihao Xuan
cs.SE, cs.AI
2026-11-16
SWE-CoWork drops coding agents into simulated 26-person engineering orgs across 101 real repos; the best models pass ~63% of F2P tests but complete under 8% of assignments.
Benchmarks in the SWE-bench family hand the agent a self-contained issue: requirements, constraints, and acceptance criteria all in the body text. Real organizations do not work that way. Requirements sit in project boards, versioned documents, chat logs, and the heads of specific colleagues, and the sources disagree: a design doc may describe a rejected approach, a green CI run may correspond to an outdated revision. The paper opens with two numbers from Anthropic: as of May 2026, over 80% of the code merged into its internal codebase was written by Claude, and a typical engineer merges 8x the lines per day compared to 2024. Code generation scaled; knowing what to build and tracking it to delivery did not. Existing benchmarks measure spec execution, not organizational work.
SWE-CoWork targets that gap: the agent joins an ongoing project as a digital employee, starts from a short handoff, and has to recover requirements, implement, verify, and deliver on its own.
Three layers.
Tasks. 101 assignments from 101 real open-source repositories, 11 languages, 12 domains. The median task adds 2,955 production lines across 60 files, graded by private executable tests: 2,699 F2P plus 562 P2P nodes across the full release.
The org. Each task ships with a 26-person engineering organization (2,626 employee profiles across the benchmark), with a median of six colleagues holding scored requirements. Every digital employee has a defined responsibility, bounded local knowledge, and fixed collaboration rules: they may demand a reproduction first, answer one topic at a time, defer a question until you follow up, or refer you to the actual owner. The environment gates what can be disclosed and when; an LLM generates the reply. All evaluated agents face the same colleagues, played by GPT-5.6 Terra.
The records. The agent starts with a median 206-word onboarding note; requirements are scattered across 10,329 board cards, 5,097 versioned documents, 40,836 chat messages, and 1,194 merge requests. The corpus plants superseded drafts, logs whose titles name the wrong run, canceled or duplicate cards, and colleagues who know an adjacent boundary but lack decision authority. Every misleading record ships with an accessible correction or a responsible owner to ask.
The central design choice is linkage: each scored requirement is tied to its holder, a discovery route, and specific test nodes, so it can be traced from the first question to the delivered code. A paired Oracle-Spec condition provides the complete requirement document upfront with no colleagues to consult, holding the repository, tests, scaffold, and 8-hour budget constant, to separate acquisition failures from implementation failures.
| Model | F2P | P2P | Pass rate |
| Claude-Opus-5.5 | 63.84% | 93.24% | 6.93% |
| GPT-6 Astra | 62.91% | 89.15% | 7.92% |
| Seed Evolving | 39.09% | 94.48% | 0.99% |
| Qwen3.8-Max | 35.70% | 91.80% | 0.00% |
| DeepSeek V4 Pro | 13.04% | 90.21% | 0.00% |
Five of eleven models shown; the other six fall between 21% and 31% F2P with at most 0.99% complete-task success.
Three numbers carry the paper. Requirement acquisition correlates with F2P at Spearman ρ = 0.94, while tool-call counts barely correlate at ρ = 0.08. GPT-6 Astra obtains 67.8% of colleague-held requirements; DeepSeek V4 Pro sends a median of zero colleague messages per run and obtains 1.8%. Second, Oracle-Spec lifts F2P by roughly 16 points for both Seed Evolving (39.09 to 55.76) and GPT-6 Astra (62.91 to 78.84), while complete-task success stays exactly where it was (0.99% and 7.92%). Handing over the spec does not finish the job. Third, the Claude-Opus-5.5 trajectory analysis: 93% of first questions go to the accountable owner, yet only 64% of colleague-held requirements are obtained. Of 450 deferred threads, 153 are never followed up, abandoned with a median 7.1 hours of budget left. Of 826 requirements reported done in handoff notes, 255 (31%) fail their tests. There are 49 premature stops with a median 2.8 hours remaining, and on the carapace-bin task the agent obtained all 25 colleague-held requirements yet still failed 22 of 35 F2P nodes.
Finding the right owner is solved; tracking the requirement to delivery is not.
This is the first benchmark to score organizational software engineering with executable tests, at per-requirement granularity: asked or not, implemented correctly or not, reported honestly or not. The diagnostics double as a design checklist for agent builders, for example an explicit per-requirement state table recording the owner, open questions, and verification evidence.
The model-selection implication is blunt. Supervised single-point coding is in decent shape: P2P sits near 90%, so existing behavior mostly survives. Handing an agent a whole assignment is not: the best models complete fewer than one task in ten. Read next to Anthropic's 80% figure, the message is that code volume is cheap and delivery management is expensive.
This is a measurement contribution, not a new method, and it is upfront about that.
The paper concedes that cross-model associations (earlier first edit, more colleague messages correlating with F2P) are not causal, that the behavioral profile relies on a rubric-based LLM reviewer and is exploratory, and it cites prior work showing simulated users differ systematically from human ones. The colleagues are rule-governed GPT-5.6 Terra personas; they never initiate contact the way real colleagues do.
Open questions remain. Oracle-Spec was run for only two models and not for the strongest one, Claude-Opus-5.5, so the claim that full specs do not rescue completion rests on a single frontier model. The pass-rate bar is brutal by construction, with a median of 32 test nodes all required to pass in one run, and with 101 tasks a 0.99% score is literally one task. The effects of the 8-hour budget and of the two scaffolds (Codex for GPT-6 Astra, Claude Code for the rest) are not ablated.