ττ-Bench: Best coding agent passes just 23.9% of realistic agent-building tasks
rohanpaul_ai · x · 2026-09-21
A new arXiv paper introduces ττ-Bench, a benchmark that makes agent construction the task itself: a developer agent receives real business records, a client holding requirements, a production API, a codebase to inherit, and cost/model limits.
Key findings:
- 53 tasks across four domains; agents are scored by deploying their customer-service agent against held-out simulated users.
- The strongest setup, Claude Opus 5 under Claude Code, passes only 23.9% of evaluations vs. an 82.2% expert-authored ceiling.
- Failures mirror human dev pitfalls: shallow queries instead of deep understanding of records, almost no client communication, and too little experimentation with architecture and serving spend.
The authors conclude that better code generation alone won't fix agents' shallow requirement comprehension.
Related event: ττ-bench Exposes Coding Agents: Top Model Passes Only 23.9%(2 posts)→
More from coding & agent
- SemiAnalysis: Persistent agents now eat 45% of our AI spend, and Perplexity's harness beats Claude Tags — dylan522p · 2026-11-12
- Ora's Ranker 3.0 grades whether agents can actually use your site; scores drop 40+ points — yenkel · 2026-11-12
- Gemini API Managed Agents now support self-hosted sandboxes, incl. Vercel — steren · 2026-11-12
- Rayline launches ARC, a per-turn model router claiming 90% cost cuts — testingcatalog · 2026-11-12
- Karpathy's llm-wiki: a pattern for building personal knowledge bases with LLM agents — stochasticchasm · 2026-11-12
- Agent self-improvement with frozen weights: failure traces add just 4pp, blind search gets 60% — burny_tech · 2026-11-12