Alibaba's Co-Design Bench: top coding agent fully completes just 26.4% of real user requests
krishnan · x · 2026-11-12
Co-Design Bench, from Alibaba and university researchers, pulls 273 tasks from 26,314 real coding sessions, graded against 2,331 hidden acceptance criteria. While top models exceed 99% on older benchmarks, the best (GPT-6 Astra) fully finishes only 26.4% of tasks, meeting 69.9% of criteria on average. Notably, one model scored 75% vs 56% on large overhauls depending on its harness—leaderboards can't capture that. The authors suggest scoring agent plus harness on your own rejected requests instead.
More from coding & agent
- NetBird routing peer lets you reach uncliented servers securely without opening ports — philipvollet · 2026-11-13
- Gemini's new Decision API tested: AI auto-orchestrates UI decision paths — huangyun_122 · 2026-11-13
- Asking an AI coding agent for a one-line install script, got a full cross-platform paragraph — gethackteam · 2026-11-13
- New Company Agent Bench compares Dots, Grok Bot, Claude Tag, Devin on company knowledge — philhchen · 2026-11-13
- Tale of "Fable 5.2": 1,000 subagents build a bootable OS in eight days — ChrisGPT · 2026-11-13
- Dev shows off parallel AI coding setup: three MacBooks running Claude Code and Codex at once — AIandDesign · 2026-11-13