SWE-CoWork: coding agents pass 63% of tests but complete fewer than 8% of tasks
Tokenwave-AI · hf · 2026-11-17
SWE-CoWork benchmarks coding agents as digital employees: 101 real assignments across 101 repos, 11 languages and 12 domains, each set in a simulated engineering org of 26 colleagues with bounded knowledge. Agents must recover requirements from handoffs, reconcile conflicting evidence, and pass private executable tests. Across 11 models, a huge gap separates partial implementation from full delivery: the two leading models satisfy 63% of fail-to-pass tests but complete fewer than 8% of assignments. Giving complete requirements upfront lifts F2P by 16 points for Seed Evolving and GPT-6 Astra yet leaves full success unchanged. Trajectory analysis shows failures throughout delivery, e.g. Claude-Opus-5.5 routes 93% of first questions to the right owner but obtains only 64% of colleague-held requirements.
More from coding & agent
- poteto on autonomous agents: 6 months of infra investment led to thousands of PRs while sleeping — 4310sy · 2026-11-17
- Tencent Hunyuan's Accelerate Bench tests 16 coding agents on 32 CPU optimization tasks — Tencent-Hunyuan · 2026-11-17
- SharePenny adds MCP support so a chat message can record and split expenses — Cultural-Effective62 · 2026-11-17
- AI engineering job-hunt workshop: scrape real JDs, build portfolio, tailor CV with an AI pipeline — Al_Grigor · 2026-11-17
- Rust MCP code-search server benchmarked against ripgrep on Grafana's 24k files — Designer-Seaweed4661 · 2026-11-17
- Verus Proof Index: new formal verification benchmark ranks 25 models, GPT-6 Astra leads at 59.1 — fhuszar · 2026-11-17