SWE-CoWork: coding agents pass 63% of tests but complete fewer than 8% of tasks

Tokenwave-AI · hf · 2026-11-17

SWE-CoWork benchmarks coding agents as digital employees: 101 real assignments across 101 repos, 11 languages and 12 domains, each set in a simulated engineering org of 26 colleagues with bounded knowledge. Agents must recover requirements from handoffs, reconcile conflicting evidence, and pass private executable tests. Across 11 models, a huge gap separates partial implementation from full delivery: the two leading models satisfy 63% of fail-to-pass tests but complete fewer than 8% of assignments. Giving complete requirements upfront lifts F2P by 16 points for Seed Evolving and GPT-6 Astra yet leaves full success unchanged. Trajectory analysis shows failures throughout delivery, e.g. Claude-Opus-5.5 routes 93% of first questions to the right owner but obtains only 64% of colleague-held requirements.

Original post →

More from coding & agent

coding & agent channel →