SWE-CoWork benchmark: best model fully passes only 8 of 101 handoff coding tasks
omarsar0 · x · 2026-11-18
TokenWave released SWE-CoWork, a benchmark evaluating coding agents as "digital employees": each assignment starts from a short handoff, and the agent must recover requirements by searching task boards and versioned docs, asking simulated teammates, and revising code when replies change the scope — delivering a patch that passes hidden tests.
Key facts and results:
- 101 real open-source repos, 11 languages, 12 domains, 26 simulated colleagues per task
- Top model GPT-6 Astra fully passed only 8/101 tasks (7.92%); Claude Opus 5.5 passed 7/101; most others scored 0–1
- The benchmark traces open questions and confirmed requirements across conversation, patch, and tests
- Sebastian Raschka (omarsar0) highlights that for Claude Opus 5.5, more than half of failed checks came after the agent already had the requirement
Takeaway: today's coding agents remain weak at taking over teammates' work-in-progress — requirement recovery and long-horizon coordination are major failure points.
Related event: SWE-CoWork Benchmark: Top Coding Agents Complete Under 8% of Tasks(2 posts)→
More from coding & agent
- LangChain hosts SF hands-on workshop on Managed Deep Agents with LangSmith — LangChain · 2026-11-18
- Vite+ 1.7.0 adds type-aware linting via tsrs, a Rust port of TypeScript 7 — cnakazawa · 2026-11-18
- Higgsfield's new AI film: 'The hardest part is making it not look like AI' — CurieuxExplorer · 2026-11-18
- Project triangle flips in the AI agent era: defining 'good' becomes the job — lak_luster · 2026-11-18
- The levels of agentic engineering: find what you still do by hand, then automate it — rseroter · 2026-11-18
- GitHub Next's Maggie Appleton on speculative design for prototyping software futures — JayAlammar · 2026-11-18