New coding-agent benchmark: Claude Opus 5 passes 23.9% vs 82.2% human expert
dair_ai · x · 2026-09-08
dair-ai highlights a benchmark paper where the developer agent must deliver a working customer-service agent inside a realistic client engagement: business records, a client holding requirements, a production API, an inherited codebase, and hard cost/model limits. Claude Opus 5 under Claude Code passes 23.9% of evals vs 82.2% expert human, across 53 tasks in four domains, scored against held-out simulated users. Failure modes: shallow record querying, telling the client almost nothing, and little experimentation with agent architecture.
More from coding & agent
- Sol61 flunks instruction-following: skips mandated research even when explicitly told — lucasmeijer · 2026-10-26
- Ben Bajarin: Frontier AI is becoming the enterprise cyber-defense stack — BenBajarin · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26