ττ-bench: Best Coding-Agent Setup Passes Just 23.9% of Real Client Simulations
rohanpaul_ai · x · 2026-09-21
The new ττ-bench treats agent building like a real client job: the model gets scattered company records, a client, an API, existing code, and a budget, then must ship an agent for unseen customer requests.
Key findings:
- The best setup, Claude Opus 5 with Claude Code, passed only 23.9% of held-out customer simulations across 53 tasks, versus 82.2% for the expert-authored reference.
- Most failures were outside pure coding: agents searched records instead of understanding them, barely questioned the client, reused familiar designs, and wrote tests that missed their own errors.
- Requirement gathering matters: on client-enabled tasks, builds asking 0 questions averaged 0.16, while those asking 4+ averaged 0.50.
The takeaway: better code generation alone won't fix this — coding agents also need to gather requirements, compare designs, use budgets intelligently, and run tests that expose their own blind spots.
Related event: ττ-bench Exposes Coding Agents: Top Model Passes Only 23.9%(2 posts)→
More from coding & agent
- Google open-sources ARTEMIS, letting AI agents control a real phone end to end — dr_cintas · 2026-09-21
- Anyone's agents actually making money? A dev's reality check on x402 agent payments — thranduilsson · 2026-09-21
- Tsinghua's DiffuTester generates unit tests with diffusion LLMs 2-3x faster — jiqizhixin · 2026-09-21
- Turning a Linux desktop into an agent workspace: Claude, Codex and Hermes on Omarchy — Teknium · 2026-09-21
- Developer builds jev, an interpreter that reasons over plain-English facts and rules, inspired by Geoffrey Litt — narphorium · 2026-09-21
- evmscope MCP server ships 20 blockchain tools for AI agents across 5 EVM chains — modelcontextprotocol · 2026-09-21