ττ-bench: Best Coding-Agent Setup Passes Just 23.9% of Real Client Simulations
rohanpaul_ai · x · 2026-09-21
The new ττ-bench treats agent building like a real client job: the model gets scattered company records, a client, an API, existing code, and a budget, then must ship an agent for unseen customer requests.
Key findings:
- The best setup, Claude Opus 5 with Claude Code, passed only 23.9% of held-out customer simulations across 53 tasks, versus 82.2% for the expert-authored reference.
- Most failures were outside pure coding: agents searched records instead of understanding them, barely questioned the client, reused familiar designs, and wrote tests that missed their own errors.
- Requirement gathering matters: on client-enabled tasks, builds asking 0 questions averaged 0.16, while those asking 4+ averaged 0.50.
The takeaway: better code generation alone won't fix this — coding agents also need to gather requirements, compare designs, use budgets intelligently, and run tests that expose their own blind spots.
Related event: ττ-bench Exposes Coding Agents: Top Model Passes Only 23.9%(2 posts)→
More from coding & agent
- Grok Bots 1.0 leaks: multiplayer teammates, proactive agents in Slack and Teams — XFreeze · 2026-11-12
- Could MCP-powered copy-trading disrupt ETFs? Investor floats paying 0.5% for trade signals — MartinGTobias · 2026-11-12
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Ten wild Gemini 4 Carbon demos in 24 hours: CUDA meadows, V8 engines, mech stomps — socialwithaayan · 2026-11-12
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Hamel Husain's complete AI evals framework for PMs and engineers — aakashgupta · 2026-11-12