Real-world coding agent benchmark: best model fully solves only 26.4% of tasks
emmanuelvivier · x · 2026-11-09
Researchers from Alibaba and several Asian universities built Co-Design Bench, testing coding agents on real user change requests — with results far below saturated old benchmarks.
- Agents must rework existing web apps from user requests, sometimes with images, audio, video or documents as input.
- The best of ten models, GPT-6 Astra, meets about 70% of a task's requirements on average but is fully correct in only 26.4% of tasks; most runs leave at least one detail unfinished, so users almost always clean up by hand.
- The authors note old benchmarks are saturated: models that run, check and fix their own code score above 99%, making model differences hard to measure; Co-Design Bench shows a gap of more than 35 percentage points versus benchmarks like WebCompass-edit.
- The dataset favors real cases where users had to rephrase requests multiple times or roll back changes.
Related event: New Benchmark Shows Top Coding Agents Fully Solve Only 26% of Tasks(2 posts)→
More from coding & agent
- Dev take: a big corporate skill library is an underrated superpower — m4rkmc · 2026-11-09
- GitNexus Turns Any GitHub Repo Into an In-Browser Interactive Knowledge Graph, Fully Open-Source — JafarNajafov · 2026-11-09
- Better Paseo plugin puts every AI agent on one task board across phone, iPad and desktop — intellectronica · 2026-11-09
- Alibaba open-sources Zvec, an in-process vector DB that's already at 16.2k GitHub stars — bigaiguy · 2026-11-09
- Keep, the save-as-Markdown tool for AI agents, launches on iOS after 2-month App Store review — iannuttall · 2026-11-09
- Claude Code cross-session messaging in parallel worktrees: two limits and a hardening config — Independent-Sir3234 · 2026-11-09