Co-Design Bench: top coding agents fully satisfy only 26.4% of real-user tasks
Qwen · hf · 2026-10-30
Qwen (Alibaba) with Fudan, Zhejiang, CUHK and others releases Co-Design Bench, a reference-free benchmark testing whether coding agents can turn real users' multimodal requests into working app revisions.
- Scale: 273 tasks curated from 37,463 deduplicated queries across 26,314 real coding sessions, paired with 2,331 hidden expert-authored acceptance criteria.
- Structure: two tracks—Multimodal Editing (localized edits with optional image/audio/video/document inputs) and Repository Editing (coordinated multi-file revisions)—across three capability dimensions: Grounded Understanding, UI Transformation, Functional Implementation.
- Protocol: judge-then-refute grades criteria from source and browser evidence, then a separate LLM verifier challenges positive verdicts—no gold code or target screenshots needed.
- Findings: the strongest of ten model endpoints passes 69.9% of criteria per task but fully satisfies only 26.4% of tasks; all endpoints drop on repository edits spanning 3-4 change types, exposing the gap between partial requirement satisfaction and complete task success.
More from coding & agent
- CodeGraph: 75.9k-star local code knowledge graph for Claude Code, Cursor & Copilot via MCP — JeremyCMorgan · 2026-10-30
- MCP Server Wraps Argentina Datos API for Holidays, FX Rates and Legislative Data — modelcontextprotocol · 2026-10-30
- Dropbox MCP Pack Wraps Dropbox API v2 for AI Assistants — modelcontextprotocol · 2026-10-30
- Video Model Shootout: Pika H3 Flex vs Three Distilled H3 Recipes on Character Continuity — rileygstaliger · 2026-10-30
- seriesmith: MIT-Licensed Tool Generates 22-Minute AI Episodes From Scripts With Character Consistency — bgstratt · 2026-10-30
- Official Claude Code VS Code Extension Trojanized: v2.1.313 Plants SSH Backdoor — jd_pressman · 2026-10-30