New Co-Design Bench: best coding agent fully completes only 26.4% of real-world tasks
GaryMarcus · x · 2026-11-12
Researchers from Alibaba and several Asian universities built Co-Design Bench, a coding-agent benchmark from real user change requests to existing web apps, with inputs including images, audio, video and documents. Key findings:
- Of ten models tested, the best (GPT-6 Astra) meets 70% of task requirements on average, but fully completes only 26.4% of tasks
- In most cases the agent does the bulk of the work but leaves at least one detail unfinished, so human cleanup is almost always needed
- Authors argue existing benchmarks are saturated: models scoring >99% when they can run, check and fix their own code; the gap vs. WebCompass-edit exceeds 35 percentage points
Gary Marcus amplified it with a jab: "AI agents still need human cleanup in roughly three out of four tasks. If only someone could have foreseen this!"
More from Models
- Unsloth's new quantization makes local Qwen Flash Next feel almost like Qwen4 — danielhanchen · 2026-11-12
- Benchmark criticized for cherry-picking known failures; same model scores 75% vs 56% across harnesses — tallmetommy · 2026-11-12
- RF-DETR Large: best OSS-licensed object detector at just 34M params — TheZachMueller · 2026-11-12
- Telling Gemini you're Karpathy makes it think 30% longer and score 7.8 vs 5.8 in blind test — dejanseo · 2026-11-12
- Continu Intel launches ci-proto: tops fev-bench with 64x context window, open weights — s_mohinii · 2026-11-12
- Reddit user finds no single LLM wins for agents: speed, accuracy, cost tradeoffs — LeadWithHitesh · 2026-11-12